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ABSTRACT 

Peters,  Benjamin,  A  Study  Comparing  the  Pedagogical  Effectiveness  of  Virtual  Worlds  and  of 
Traditional  Training  Methods.  Master  of  Science  (MS),  August,  2014,  106  pp.,  1 1  tables,  59 
figures,  references,  41  titles. 

This  experiment  tests  whether  a  virtual  world  is  a  more  suitable  alternative  to  classical 
paper  and  pen  case  studies  for  teaching  people  how  to  disann  improvised  explosive  devices 
(IED).  Following  preliminary  training,  the  subjects  are  separated  into  a  group  of  32  and  one  of 
33.  The  odd-numbered  subjects  receive  case  studies  while  the  even-numbered  subjects  receive 
training  in  the  virtual  world,  Second  Life.  After  their  training,  each  subject  is  put  into  a  simulated 
test  where  they  attempt  to  properly  disann  a  mock  IED  attached  to  a  victim  played  by  an 
actor/actress.  The  results  of  the  experiment  show  no  significant  difference  between  the  two 
instruction  types  in  terms  of  the  subjects’  ability  to  perfonn  the  correct  procedures  in  a  situation 
with  an  IED.  However,  a  higher  percentage  of  subjects  taking  the  Second  Life  training  properly 
disarmed  the  IED  than  that  of  those  taking  the  case  studies. 
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CHAPTER  I 

INTRODUCTION 

1.1  Virtual  Reality 

Virtual  reality  is  a  fairly  new  form  of  technology  finding  its  way  into  the  world  of 
business,  government,  and  other  organizations.  While  it  has  been  a  staple  of  the  entertainment 
industry,  virtual  reality  has  struggled  to  find  favor  when  used  as  a  training  tool.  Due  to  cost, 
skepticism,  or  a  resistance  to  change,  organizations  have  been  slow  to  embrace  using  virtual 
reality  for  training.  With  the  creation  of  Linden  Lab’s  Second  Life  in  2003,  the  perception  of 
virtual  reality  training  has  improved.  Expensive  training  tools  have  been  replaced  by  digital 
replicas  in  a  virtual  island.  Through  the  proper  programming,  these  replicas  work  like  their  real- 
world  counterparts,  but  they  lack  the  safety  concerns  involved  in  having  an  unskilled  operator 
using  potentially  dangerous  machinery.  However,  there  are  concerns  about  whether  this  new 
technology  is  more  effective  than  traditional  training  methods  at  training  operators.  These 
concerns  are  valid  since  making  less  effective  changes  would  be  costly  to  an  organization. 
Therefore,  proponents  and  skeptics  alike  need  concrete  evidence  to  determine  if  in  fact  virtual 
reality  is  more  effective  at  training  people  than  traditional  training  methods. 

The  factors  playing  a  central  role  in  the  effectiveness  of  virtual  reality  training  are 
presence  and  immersion.  Presence  refers  to  a  person’s  feeling  of  being  physically  present  in  a 
virtual  environment  even  when  in  reality  they  are  not  physically  present.  The  level  of  presence 
depends  on  the  level  of  realism  in  the  environment.  However,  even  if  the  environment  simulates 
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real  life  perfectly,  it  can  be  useless  as  a  training  tool  unless  the  trainee  is  immersed  in  the 
environment.  Immersion  refers  to  how  involved  a  person  is  with  an  environment  whether  the 
environment  is  real  or  synthetic.  Asking  if  a  person  is  immersed  in  an  environment  is  the  same  as 
asking  if  the  person  believes  that  they  are  in  that  environment.  Therefore,  when  designing  a 
training  exercise  it  is  important  to  make  sure  the  exercise  holds  the  interest  of  those  being 
trained.  The  more  interested  a  person  stays  in  the  exercise,  the  more  effective  it  should  be.  Over 
the  years,  different  companies  have  attempted  to  use  these  concepts  to  introduce  innovative 
training  exercises.  While  not  perfect,  they  have  laid  the  groundwork  for  future  projects 
attempting  to  determine  the  effectiveness  of  virtual  training. 

The  types  of  virtual  reality  experiments  conducted  range  from  simple  online  instructional 
videos  to  fully  developed  3D  virtual  rooms.  These  experiments  were  developed  with  the  thought 
of  improving  the  more  traditional  styles  of  training  workers,  athletes,  or  military  personnel. 

While  traditional  training  methods  might  be  adequate  for  certain  types  of  work,  such  as  welding 
or  most  athletic  sports,  they  have  the  potential  to  become  too  costly,  or  even  too  dangerous  for 
the  trainees.  Take  the  welding  example  for  instance.  The  metal  being  welded  in  a  training 
exercise  is  still  useable  metal.  If  the  weld  is  prepared  poorly,  then  the  metal  is  wasted  and  cannot 
be  reused.  If  in  some  way  a  virtual  training  tool  is  developed,  then  no  resources  would  be  wasted 
during  training.  The  training  is  determined  to  be  effective  if  the  welder  can  provide  as  good  of  a 
weld  as  someone  who  was  trained  traditionally.  This  example  shows  the  rationale  used  by 
companies  when  developing  their  own  innovative  training  methods. 

Innovation  in  training  now  is  rooted  in  using  the  computer  as  a  powerful  tool  to  train 
workers  in  a  variety  of  different  ways.  The  simplest  examples  are  programs  where  a  trainee  sits 
and  watches  an  online  tutorial  of  how  to  do  something.  The  trainee  is  quizzed  following  the 
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tutorial  to  see  what  they  have  learned.  Slightly  more  immersive  are  third-person  simulations. 
These  are  simulations  where  trainees  take  control  of  avatars  in  order  to  perfonn  a  specific  task. 
One  example  is  the  use  of  a  simulation  to  train  people  for  emergency  evacuations  of  buildings.  In 
order  to  add  some  realism  to  the  training,  smoke  and  vibrations  can  be  used  to  simulate  the 
stages  of  a  collapsing  building. 

The  level  of  immersion  increases  in  simulations  where  sensors  are  attached  to  the 
trainees.  These  sensors  correspond  to  an  avatar  on  the  computer  that  follows  the  movements  of 
the  trainee.  The  trainee  can  move  their  hands  in  a  matter  similar  to  movement  they  would  use  if 
they  were  perfonning  a  particular  task,  say  carpentry.  In  this  example  a  block  of  wood  is 
positioned  on  a  table  in  a  virtual  world.  By  moving  the  hand  avatar  over  the  wood,  the  trainee 
can  hold  the  wood  in  place  while  sawing  it  using  their  other  hand.  They  do  this  by  grabbing  a 
virtual  saw  and  moving  their  arm  in  a  motion  identical  to  how  they  saw  in  real  life. 

3D  simulations  are  the  most  advanced  when  it  comes  to  technology.  These  are  built  by 
designing  3D  virtual  environments  that  a  trainee  can  step  into  and  interact  with  a  simulated 
environment.  Through  the  use  of  projectors  and  mirrors,  walls  of  3D  images  surround  the  trainee 
creating  the  mirage  of  a  different  world  designed  to  the  creator’s  choosing.  However,  even  with 
all  these  advances,  one  question  still  persists.  How  is  the  effectiveness  of  these  trainings 
determined? 

1.2  Assessment  Tools 

In  addition  to  creating  these  different  training  tools,  researchers  have  used  different 
methods  to  detennine  if  their  tools  are  effective  or  not.  These  methods  lean  towards  using 
subjective  questionnaires.  The  trainees  are  given  questionnaires  to  fill  out  following  the 
conclusion  of  their  training.  These  questionnaires  tend  to  use  5-point,  7-point,  or  10-point  scales 


26 


for  each  question.  The  points  are  tallied  and  the  results  are  reported.  Often,  the  trainees  report 
they  liked  the  training  and  felt  a  sense  of  presence.  A  more  sophisticated  way  of  reporting  results 
comes  from  using  statistical  methods.  Statistics  are  taken  on  the  sample  itself  for  more  detailed 
information  on  the  people  taking  the  training.  Age,  gender,  history  with  virtual  reality,  history 
with  the  company,  etc.  are  all  reported.  The  results  of  the  trainings  are  reported  as  statistical 
means  and  variances  and  through  hypothesis  testing,  they  are  compared  to  a  control  sample. 
However,  what  are  lacking  are  any  physiological  studies  on  the  trainees.  While  there  are  studies 
about  how  the  body  reacts  to  stimuli  when  it  is  immersed  in  an  environment,  virtual  trainings 
tend  not  to  include  this  type  of  study.  In  order  to  detennine  the  effectiveness  of  virtual  reality 
training,  a  quantitative  method  for  detennining  presence  and  immersion  needs  to  be  used. 
Overall,  the  research  done  on  virtual  reality  for  training  purposes  is  lacking  because  this  type  of 
investigation  has  been  omitted. 

1.3  Purpose  of  Experiment 

The  purpose  of  this  experiment  is  to  detennine  whether  virtual  reality  training  is  more 
effective  for  training  human  subjects  than  traditional  training  methods.  As  previously  stated, 
concrete  evidence  is  needed  to  affinn  or  refute  this  hypothesis.  Evidence  is  gathered  by 
perfonning  an  experiment  using  human  subjects.  The  subjects  are  split  into  two  groups.  Both 
groups  are  trained  in  defusing  an  explosive  device.  One  group  receives  virtual  reality  training 
and  the  other  receives  only  the  traditional  style  training.  The  subjects  are  tested  in  defusing  a 
mockup  of  a  bomb  in  a  live  simulation.  They  are  timed  and  these  times  determine  the  success  or 
failure  of  the  virtual  training  procedure.  In  addition,  the  subjects’  physiological  responses  are 
monitored  during  the  final  day  of  the  experiment  in  order  to  detennine  if  there  is  a  difference  in 
the  responses  for  the  two  instruction  types.  In  other  words,  the  feeling  of  presence  and 
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immersion  of  the  subject  in  the  virtual  world  is  being  tested  scientifically  and  not  just  by  a 
qualitative  questionnaire.  Through  these  methods  the  hypothesis  that  virtual  training  is  more 
effective  than  a  traditional  training  method  is  evaluated. 

1.4  Outline  of  Report 

Before  diving  into  the  experiment  though,  an  extensive  review  on  the  literature  covering 
the  topics  of  virtual  reality  training,  sense  of  presence  during  training,  and  training  effectiveness 
evaluation  is  provided  in  Chapter  2.  This  literature  review  covers  previous  research  undertaken 
in  order  to  provide  a  good  picture  of  what  has  been  accomplished  in  the  field  of  virtual  reality 
training.  It  also  provides  infonnation  on  how  particular  training  methods  are  evaluated.  These 
evaluation  tools  are  useful  for  creating  a  plan  on  how  to  evaluate  the  experiment  highlighted  in 
this  thesis.  As  will  be  seen,  some  of  these  methods  rely  primarily  on  qualitative  analysis. 
However,  there  are  a  few  experimenters  which  use  quantifiable  data  in  their  evaluation  methods. 

In  Chapter  3,  the  experiment  upon  which  this  report  is  based  is  explained  in  detail.  This 
includes  a  summary  of  how  the  subjects  are  separated  into  two  different  classes  -  one  which 
trains  the  subjects  in  the  defusing  of  a  bomb  using  virtual  reality  training  and  another  which 
trains  the  subjects  using  traditional  training  methods.  The  testing  simulation  the  subjects 
undertake  is  explained  in  detail  and  the  tools  by  which  the  data  is  gathered  are  explained. 
Included  in  this  explanation  is  a  description  of  the  arrangement  of  physiological  sensors.  Chapter 
4  focuses  on  the  analysis  of  the  data.  It  gives  a  full  analysis  on  what  the  results  of  the  experiment 
are.  This  provides  an  answer  to  the  question  of  whether  virtual  reality  is  more  effective  than 
traditional  methods.  Chapter  5  serves  as  a  conclusion  to  the  thesis.  It  summarizes  the  results  from 
the  experiment  and  explains  the  impact  of  these  findings. 
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CHAPTER  II 

LITERATURE  REVIEW 

2.1  Introduction 

The  use  of  virtual  reality  as  a  tool  for  training  purposes  has  been  studied  since  the  latter  half  of 
the  twentieth  century.  Unfortunately,  the  conclusions  linked  to  these  studies  tend  to  say  that  the 
research  done  on  virtual  reality  is  inadequate.  Dickey  (2003)  claims,  “more  research  needs  to  be 
done  to  fully  explore  the  potential  [of  virtual  reality.]”  When  it  comes  to  the  topic  of  virtual 
reality,  Gaimster  (2008)  says  there  has  been  “Little  research  in  this  specific  area  . ...”  Goel 
(2009)  supports  this  assertion  by  saying,  “There  is  little  research  that  addresses  what  features  of 
virtual  worlds  support  ...  applications.”  While  much  of  the  research  is  in  need  of  further  testing, 
there  have  been  major  advancements  in  the  use  of  virtual  reality  for  training  purposes.  These 
advancements  are  backed  with  statistical  proof  of  virtual  reality’s  effectiveness. 

This  chapter  is  devoted  to  literature  on  virtual  reality  technology.  While  some  of  the 
projects  presented  may  not  be  completed,  the  chapter  still  provides  an  idea  of  what  has  been 
done  so  far.  It  also  persuades  readers  to  see  the  benefits  of  using  virtual  reality.  But  before 
moving  to  the  experiment,  a  review  on  the  literature  about  virtual  reality  needs  to  be  explored. 
The  review  includes  an  overview  of  virtual  reality  and  critical  thinking  and  learning.  Included  in 
the  overview  are  projects  which  support  the  use  of  virtual  reality  and  its  effect  on  learning. 
Following  the  overview,  the  ideas  of  presence  and  immersion  are  discussed  with  articles  related 
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to  these  ideas.  Since  presence  and  immersion  tie  into  physiological  response,  a  section  is 
dedicated  to  that  topic.  Afterwards,  there  is  a  section  dedicated  to  the  types  of  experiments  run 
and  the  corresponding  results  of  those  experiments.  Finally,  projects  still  in  progress  are 
discussed  and  the  chapter  is  completed  with  a  conclusion. 

When  virtual  reality  or  virtual  worlds  are  mentioned,  the  thought  that  comes  to  mind  is 
video  games.  While  video  games  have  been  created  using  virtual  worlds  to  represent  towns, 
forests,  oceans,  etc.,  this  study  focuses  on  the  use  of  virtual  reality  for  training  purposes,  not 
gaming.  While  video  games  could  be  used  to  create  a  training  program,  this  examination  dives 
deeper  than  just  a  3D  video  game.  This  is  about  detennining  if  virtual  reality  can  simulate 
presence  for  a  person  who  is  not  actually  in  the  environment  in  which  they  see  themselves.  Then, 
once  knowing  the  person  is  immersed  in  a  virtual  reality,  whether  this  reality  can  be  used  to  train 
them  for  a  task  better  than  traditional  training  methods  needs  to  be  determined.  Before  getting 
into  that,  the  question,  “What  is  Virtual  Reality?”  must  be  asked.  Liu  and  Hao  (2004)  describe 
virtual  reality  as  a  powerful  technology  for  creating  an  interactive  virtual  environment  for  the 
purpose  of  education  and  training.  According  to  Cheng  et  al.  (2010),  virtual  reality  can  improve 
learning  performance  by  offering  hands-on  experience.  As  a  matter  of  fact,  Kelly  and  Cheek 
(2008)  looked  to  revolutionize  the  way  people  learn  using  virtual  worlds.  Their  goal  was  to 
create  a  market  for  virtual  training  in  the  future.  Their  idea  to  complete  this  goal  was  to 
“collaboratively  build  and  test  a  meta-layer  compatible  with  a  subset  of  leading  virtual  world 
platforms  that  provides  the  robust  administrative  tools  necessary  for  adoption  in  educational  and 
work  settings.”  In  other  words,  they  wanted  to  use  virtual  worlds  as  a  teaching  tool  for  students 


and  workers. 
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This  use  of  virtual  reality  is  fairly  recent.  In  fact,  virtual  reality  is  a  fairly  recent  concept. 
Holm  and  Priglinger  (2008)  trace  the  history  of  virtual  reality  as  a  technology  to  Ivan  Sutherland 
who  wrote  The  Ultimate  Display  in  1965.  This  resulted  in  the  first  virtual  reality  system:  The 
Sword  of  Damocles.  The  first  head  mounted  display  was  then  developed  in  1970.  Since  then,  a 
multitude  of  simulators,  games,  and  training  devices  have  been  created.  One  of  the  most  popular 
versions  of  virtual  reality  is  the  use  of  virtual  worlds.  “Virtual  worlds  are  computer-based 
simulated  environments  designed  to  allow  users  to  inhabit  and  interact  via  avatars,  the  human 
agent’s  in-world  representative”  (Monahan  et  al.  2009).  One  of  the  most  prominent  virtual 
worlds  is  Second  Life.  Second  Life  is  a  3D  virtual  world  created  by  Linden  Labs.  This  virtual 
world  consists  of  “islands,”  servers  that  customers  or  private  organizations  can  customize  to  how 
they  see  fit  (Heiphetz  and  Woodill  2010).  While  Second  Life  has  been  used  for  gaming,  it  also 
has  been  used  for  marketing,  classroom  instruction,  and  training  simulations.  When  it  comes  to 
education  over  1500  universities  are  using  Second  Life  (Ondrejka  2007).  As  far  as  training  goes, 
companies  believe  using  virtual  reality  for  training  is  an  efficient  way  to  teach  their  employees. 

2.2  Critical  Thinking  and  Learning 

The  perceived  strength  of  virtual  reality  is  that  it  fosters  an  environment  for  learning  and 
critical  thinking.  Scriven  and  Paul  (1992)  define  critical  thinking  as  the  “intellectually 
disciplined  process  of  actively  and  skillfully  conceptualizing,  applying,  analyzing,  synthesizing, 
and/or  evaluating  information  gathered  from,  or  generated  by,  observation,  experience, 
reflection,  reasoning,  or  communication,  as  a  guide  to  belief  and  action.”  Scriven  and  Paul  go  on 
to  say  that  critical  thinking  is  based  on  two  components:  a  set  of  skills  to  process  and  generate 
infonnation  and  beliefs,  and  the  habit  of  using  those  skills  to  guide  behavior.  While  technical 
skills  are  necessary  to  perform  tasks,  a  worker  with  good  technical  skills  and  strong  critical 
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thinking  skills  can  stand  out  against  a  worker  with  only  great  technical  skills.  This  idea  can  also 
pertain  to  combat  situations.  According  to  Hammond  (2004),  “Even  in  combat,  how  well  you 
think  is  more  important  to  how  well  you  fight  than  how  physically  fit  you  are.  A  wrong 
decision,  an  unasked  question,  a  forgotten  task,  an  incomplete  analysis,  or  a  poor  synthesis  can 
kill  you.” 

In  order  to  prevent  people  from  hanning  themselves  in  potentially  dangerous  activities 
due  to  a  lack  of  critical  thinking  skills,  virtual  reality  projects  have  been  designed  to  foster 
learning  and  critical  thinking.  Interactive  tutorials  are  a  common  usage  for  virtual  reality.  These 
tutorials  start  by  walking  the  user  through  a  set  of  instructions  and  then  allow  the  user  to  practice 
using  whatever  tools  they  need  to  master.  Three  tutorials  include:  the  Oil-field  Safety  Operation 
Training  Interactive  Virtual  Environment,  the  Virtual  Assembly  System  on  Automobile  Engines, 
and  a  tutorial  to  teach  hand  hygiene  to  hospital  employees. 

Liu  and  Hao  (2004)  designed  a  virtual  environment  called  the  Oil-field  Safety  Operation 
Training  Interactive  Virtual  Environment  (OSOTIVE).  In  this  environment,  the  trainee  goes 
through  a  series  of  levels,  or  modes,  which  increase  in  difficulty.  The  first  mode  is  the  Close 
Demonstration  Mode  (CDM).  In  this  mode,  the  trainee  is  given  a  situation  and  has  the  system 
demonstrate  the  correct  steps  to  perform  in  that  situation.  The  second  mode  is  the  Guided 
Operation  Mode  (GOM).  In  this  mode,  the  trainee  controls  an  avatar,  their  representative  in  the 
virtual  world,  to  complete  the  steps  of  a  particular  job.  However,  the  program  rejects  any  wrong 
moves  so  the  user  knows  when  they  make  a  mistake.  Finally  the  third  and  final  mode  is  the 
Operation  Mode  (OOM).  In  this  mode,  the  trainee  is  allowed  to  explore  freely  without 
interruption.  If  they  make  a  mistake,  the  system  reacts  accurately  to  the  situation.  The  mode 
encourages  the  trainee  to  leam  from  their  mistakes. 
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Just  like  OSOTIVE,  the  Virtual  Assembly  System  on  Automobile  Engines  (AEVAS) 
uses  a  level  system.  Designed  by  Cheng  et  al.  (2010),  AEVAS  consists  of  four  rooms: 
Knowledge  Room,  Assembly  Room,  Expert  Room  and  Checking  Room.  The  Knowledge  Room 
is  the  first  room  the  trainee  sees  after  logging  in.  They  can  view  an  assortment  of  material  on  the 
engine  parts.  The  purpose  is  to  gain  knowledge  of  the  different  parts  to  help  in  the  later  rooms. 
The  Assembly  Room  follows  and  consists  of  four  inner-rooms:  Crankshaft  Assembly  Room, 
Head  Assembly  Room,  Tensioner  Assembly  Room  and  the  Whole  Engine  Assembly  Room.  In 
each  room,  there  is  an  exhibition  of  an  assembly  for  each  section.  With  this  knowledge,  the 
trainee  is  ready  to  try  to  assemble  the  engine  in  the  next  room.  The  Expert  Room  gives  the 
trainee  the  chance  to  test  what  they  learned.  The  trainee  can  assemble  parts  in  the  virtual 
environment  and  gain  feedback  on  whether  they  did  well  or  not.  The  final  room  serves  as  a 
bonus  room.  The  Checking  Room  removes  a  part  of  the  engine  and  tests  the  trainee  to  see  if  they 
know  where  it  goes  and  the  steps  to  get  it  there.  Wen  et  al.  (2009)  created  a  similar  environment 
fit  with  an  oil  drilling  rig.  The  trainee  can  see  the  rig  from  a  3D  perspective  and  leam  how  to 
operate  it. 

Similarly  to  the  previous  three  groups,  Bertrand  et  al.  (2010)  created  a  virtual  training 
tutorial  system.  In  this  case,  they  designed  a  tutorial  to  train  hospital  workers  in  proper  hand 
hygiene  procedures.  In  the  virtual  training,  they  introduce  five  times  when  employees  should 
wash  their  hands.  Those  are: 

•  Before  touching  a  patient. 

•  Before  clean/aseptic  procedures. 

•  After  body  fluid  exposure/risk. 

• 


After  touching  a  patient. 
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•  After  touching  the  patient’s  surroundings 

There  are  three  levels,  called  “phases”  in  the  simulation.  The  first  is  the  Tutorial  Phase.  Here  the 
trainee  is  given  an  overview  of  the  five  moments  for  hand  hygiene  by  a  virtual  doctor  named  Dr. 
Evan.  The  doctor  prepares  the  trainee  by  telling  them  that  they  are  to  be  tested  in  the  five 
moments  of  hand  hygiene.  The  doctor  demonstrates  the  situations  and  uses  voice  and  expression 
to  do  so.  The  Interactive  Training  Phase  shows  a  virtual  nurse  named  Simon  who  interacts  with 
patients.  The  trainee  must  answer  whether  or  not  Simon  followed  the  correct  hand  hygiene 
procedures.  Finally  in  the  Feedback  Phase,  the  trainee  is  scored  on  how  well  they  answered  the 
questions.  It  should  be  noted  this  is  not  a  very  immersive  system  as  the  trainee  does  not  interact 
through  an  avatar. 

A  similar  scoring  method  is  used  in  a  driving  simulator  (Fiang  2007).  The  simulator  is 
created  as  a  project  for  a  Computer-Aided  Design  class.  The  simulator  allows  the  trainee  to  take 
a  simulated  driving  test.  For  every  mistake  the  trainee  makes,  their  score  is  reduced  by  a  certain 
amount.  If  the  trainee  drops  below  70  points  the  simulation  ends  and  is  reset.  Consequently,  the 
trainee  passes  the  test  if  their  score  is  over  70  after  they  complete  the  exam.  Fiang  is  not  unique 
in  his  idea  to  use  a  scoring  system  in  a  simulator.  The  idea  of  using  feedback  to  aid  in  a 
simulation  is  also  used  by  Marcos  de  Moraes  and  dos  Santos  Machado  (2009)  in  a  bone  marrow 
harvest  simulator. 

In  their  book,  Training  and  Collaboration  with  Virtual  Worlds,  Heiphetz  and  Woodill 
(2010)  support  the  use  of  Second  Fife  by  referring  to  two  particular  success  stories.  The  first 
story  is  that  of  Michelin.  Michelin  had  been  dealing  with  increased  competition  and  therefore 
created  the  foundation  of  a  global  delivery  model  for  information  systems.  In  order  for  this  to 
work,  they  needed  to  train  200  workers  across  three  continents.  First,  Michelin  tried  using 
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traditional  training  methods,  but  they  failed.  As  a  result,  Michelin  looked  to  use  virtual 
technology  to  train  their  employees.  They  settled  on  Second  Life  due  to  Second  Life’s  offering 
of  a  virtual  classroom  and  an  arena  for  the  participants  to  train  further.  To  start  off  the  training, 
the  workers  took  a  virtual  class  where  they  had  pre-made  avatars.  After  the  class,  the  trainees 
used  their  knowledge  to  create  proposals  for  potential  business  targets  without  exceeding  budget 
constraints.  The  training  was  considered  a  success  due  to  a  low  cost  of  about  $100,000  and 
taking  much  less  time  than  the  traditional  training  methods.  In  the  other  example,  the  Kansas 
University  Medical  Center  (KUMC)  developed  a  virtual  training  simulator  for  training  nurses  in 
an  induction  sequence.  The  training  focused  on  the  sequence  of  events  of  preparing  a  patient  for 
surgery.  While  the  training  was  supposed  to  teach  the  students  the  sequence  of  events  of  the 
induction  process,  an  unexpected  benefit  also  occurred.  Since  the  hospital  room  was  modeled 
accurately,  the  students  did  not  just  learn  the  processes  of  induction,  they  learned  about  the 
layout  of  the  hospital.  The  students  reported  they  felt  the  training  was  very  effective,  and  they 
felt  the  training  would  be  useful  for  future  students. 

2.3  Presence,  Immersion,  and  Engagement 
Those  simulations  are  beneficial  for  helping  workers  leam  how  to  use  tools  or  just  to 
follow  instructions  in  general.  They  combine  uses  of  levels  which  increase  in  difficulty  with  a 
feedback  scoring  system  to  aid  in  the  trainees’  learning.  But,  trainees  need  more  than  just  a 
simulation  to  really  learn  using  virtual  reality.  If  a  user  of  these  simulations  does  not  feel 
immersed  in  the  environment  and  only  sees  it  as  a  simple  game  they  may  not  leam  from  it. 
However,  if  the  trainee  is  able  to  block  out  their  surroundings  and  hone  in  on  the  training  they 
are  going  through,  their  chances  of  learning  from  it  increases  dramatically.  This  section  focuses 
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on  this  idea  of  feeling  immersed  in  the  environment.  It  discusses  the  ideas  of  presence  and 
immersion  and  how  these  ideas  can  aid  in  learning. 

Witmer  and  Singer  (1998)  define  presence  in  virtual  worlds  as  “the  sense  of  being  there 
(in  a  virtual  environment),  even  when  one  is  physically  situated  in  another  place.”  When  training 
someone,  it  is  important  that  those  being  trained  feel  a  sense  of  presence.  Unfortunately,  the 
sense  of  presence  appears  to  be  a  very  subjective  feel  which  is  based  on  the  opinions  of  the 
trainees.  Sanchez-Vives  and  Slater  (2005)  reported  that  this  view  of  presence  as  being  subjective 
has  led  to  the  widespread  use  of  self-report  of  user  experience.  The  danger  with  this  type  of 
reporting  is  that  a  tester  has  no  way  of  knowing  whether  or  not  the  trainee  is  being  truthful.  This 
leads  to  skepticism  of  the  effectiveness  of  a  test  and  of  virtual  reality  training  in  general. 
Sanchez-Vives  and  Slater  (2005)  help  with  this  problem  by  offering  an  idea  of  when  presence 
occurs.  According  to  them,  “Presence  occurs  when  there  is  a  successful  substitution  of  real 
sensory  data  by  computer  generated  sensory  data,  and. .  .the  person  responds  to  the  virtual  stimuli 
as  if  they  were  real.”  The  ways  to  test  for  this  is  discussed  in  the  next  section.  For  now,  a  project 
dealing  with  presence  is  discussed. 

One  type  of  virtual  reality  project  related  to  the  idea  of  presence  is  a  virtual  reality 
environment  used  for  oil-field  safety  training.  Liu  and  Hao  (2004)  designed  the  world.  One 
particularly  interesting  feature  is  their  use  of  a  disembodied  hand  as  an  avatar.  The  hand  is  seen 
as  if  it  belongs  to  the  trainee,  as  the  trainee  sees  the  hand  and  the  world  from  a  first-person  point 
of  view.  The  trainee  chooses  from  several  options  of  hand  shapes,  the  best  hand  shape  to  grab  the 
tool  they  needs.  Table  2. 1  shows  the  different  hand  shapes  and  a  description  of  each  one. 


36 


Table  2.1:  Hand  Shapes 


Number 

Icon 

Description 

1 

Pinch  small  object  with  two  fingers 

2 

Pinch  bigger  object  with  more  fingers 

3 

Grasp  small  diameter  bar  or  stick  object 

4 

Grab  bigger  diameterbar  or  stick  object 

5 

Grasp  circular  object  with  palm 

. 

Pinch  small  circular  object  with  all 
fingers 

7 

Hook  object  with  fingers 

. 

Press  or  smear  something,  pressinga 
button,  flipping  a  switch 

This  attention  to  detail  is  important  for  simulating  the  methods  of  using  small  tools  that 
workers  use  in  real  life.  While  the  necessity  of  this  amount  of  detail  depends  on  what  type  of 
simulation  is  being  done,  the  thought  is  the  trainee  must  feel  more  excited,  and  therefore 
immersed,  by  the  detailed  hand  grasps  presented  in  Table  2.1. 

Immersion  plays  a  key  role  in  how  much  presence  a  trainee  feels  while  they  are  training. 
A  couple  common  tools  used  to  immerse  trainees  are  head-mounted  displays  (HMD)  and 
immersive  rooms.  As  previously  stated,  the  former  were  introduced  in  the  1970’s.  However  their 
use  has  not  gone  away.  Holm  and  Priglinger  (2008)  have  designed  a  simulator  which  involves  an 
HMD  for  refinery  workers.  The  training  consists  of  two  computers,  one  which  projects  the 
virtual  environment  to  the  HMD  and  another  which  is  controlled  by  the  tester.  The  tester  can 
trigger  occurrences  to  the  trainee  forcing  them  to  react.  The  tester  can  then  assess  the 
performance  of  the  trainee. 

The  immersive  room  is  a  more  recent  idea  being  used.  This  consists  of  a  room 
surrounded  by  monitors  or  screens  which  project  images  giving  the  trainee  the  feel  that  they  are 
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in  a  room  that  they  are  not  actually  in.  The  trainee  is  surrounded  by  projections  which  are 
supposed  to  give  the  feeling  of  presence.  The  Virtual  Environment  Radiotherapy  Training 
(VERT)  is  an  award  winning  innovation.  VERT  consists  of  an  auditorium  with  a  large  screen 
that  projects  the  images  to  the  trainee.  The  trainee  can  walk  on  the  stage  and  perform  tasks  which 
are  assigned  as  part  of  the  training. 

A  huge  benefit  of  an  immersive  environment  like  the  one  proposed  by  VERT,  is  the 
ability  to  train  as  long  as  you  want  without  endangering  someone’s  life.  VERT  is  meant  to  be  a 
training  tool  for  radiotherapy.  When  dealing  with  medicine,  mistakes  can  be  deadly.  Therefore, 
the  ability  to  train  without  worrying  about  mistakes  being  critical  is  very  useful  for  selling  the 
idea  of  virtual  reality-based  training.  As  Liang  (2007)  says,  “Immersive  VR  is  able  to  provide  a 
rich,  interactive  and  engaging  training  context  that  in  reality  would  be  too  dangerous,  too 
expensive  or  simply  impossible  to  access.”  Nowhere  is  this  more  evident  than  in  an  evacuation 
setting.  Orr  et  al.  (2008)  created  a  simulation  for  mine  evacuation.  The  simulation  included 
“smoke  that  significantly  obscures  the  trainees’  VR  vision  in  some  areas”  and  moments  when  the 
ground  would  collapse  under  the  avatars’  feet.  Obviously,  this  type  of  training  would  be 
unrealistic  in  real  life  but  in  a  virtual  environment  it  is  very  plausible. 

While  the  previous  simulations  focus  on  visual  and  audio  feedback  to  give  the  feeling  of 
presence,  Ruffaldi  et  al.  (2009)  expands  on  those  senses  by  adding  an  element  of  touch.  The 
training  simulation  is  meant  for  competitive  rowers.  The  rowers  train  by  using  oars  in  a 
simulated  environment.  While  the  trainee  is  rowing,  large  fans  are  used  to  simulate  the  force 
feedback  that  would  be  felt  if  rowing  in  the  water.  This  way,  the  trainee  can  practice  in  an 
environment  that  is  as  similar  as  possible  to  rowing  in  actual  water.  The  effectiveness  of  this 
training  and  the  other  ones  like  it  is  dependent  on  how  engaged  the  trainee  is  in  the  training. 
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Engagement  is  a  feeling  one  gets  while  interacting  with  someone  or  something.  If 
presence  is  the  “sense  of  being  there,”  and  immersion  is  interacting  in  a  world  while  losing  sense 
of  where  one  is,  then  engagement  occurs  when  one  feels  so  involved  in  a  simulation  that  they 
feel  a  sense  of  control  of  the  simulation  and  also  lose  track  of  time  (Cooper  2010).  In  Karen 
Cooper’s  article  “Go  with  the  Flow,”  Cooper  references  two  authors  to  define  engagement.  Astin 
(1984)  defines  engagement  as  the  amount  of  physical  and  psychological  energy  that  the  student 
devotes  to  the  academic  experience.  Homik  (2008)  says  student  engagement  may  be  associated 
with  increased  time  on  task,  and  the  development  of  deep  learning,  resulting  in  better  classroom 
perfonnance.  While  the  relationship  between  engagement  and  immersion  with  presence  and 
learning  cannot  be  overstated,  it  is  important  to  know  how  much  of  these  factors  does  it  take  to 
produce  an  adequate  amount  of  presence.  Bowman  and  McMahan  (2007)  advocate 
“investigating  multiple  components  of  immersion  simultaneously  with  multiple  levels  per 
component  while  still  maintaining  a  high  degree  of  experimental  control.”  According  to  that 
logic,  designing  a  highly  interactive  training  method  yields  the  best  learning  results.  However, 
there  is  a  need  for  tests  which  give  credibility  to  the  effectiveness  of  a  training  tool  and  to  the 
notion  that  the  trainees  feel  a  sense  of  presence.  The  next  section  mentions  several  tests 
perfonned  to  give  credibility  to  the  effectiveness  of  their  respective  projects. 

2.4  Testing 

In  the  previous  sections,  several  projects  related  to  virtual  reality  have  been  discussed. 
However,  these  projects  tend  to  be  experimental  in  that  they  are  innovations  which  are  believed 
to  be  helpful  in  aiding  in  learning.  Unfortunately,  they  lack  data  to  support  their  assumptions. 
This  section  includes  projects  completed  using  objective  measures.  They  fall  in  one  of  three 
categories:  comparisons  of  two  types  of  tools,  questionnaires,  and  statistical  test  results. 
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Comparisons  are  an  important  aspect  of  detennining  the  effectiveness  of  a  new 
innovation  at  least  in  a  relative  sense.  In  order  to  convince  a  market  that  it  should  use  a  different 
technology  than  it  has  been  using  for  a  long  time,  the  market  needs  to  see  that  there  is 
improvement  from  the  old  way.  The  best  way  to  do  this  is  to  use  Hypothesis  testing  which  can  be 
used  to  determine  whether  there  is  a  significant  improvement  by  a  new  technology  over  an  old 
technology.  An  example  of  this  is  by  Gruchalla  (2004)  who  compared  the  effectiveness  of  the 
task  of  oil  well  path  tracking  using  a  desktop  computer  with  that  of  a  CAVE.  Despite  his 
attempts  to  make  the  two  mediums  the  same  in  respects  to  the  program  and  screen  resolution,  the 
task  was  performed  better  in  the  CAVE  technology. 

Sometimes  though,  while  technology  seems  like  it  should  make  things  better,  the  results 
do  not  turn  out  to  be  what  was  originally  expected.  Datey  (2001)  also  used  a  desktop  in  a 
comparison.  In  this  case,  he  was  comparing  it  to  an  HMD.  The  test  was  for  infonnation 
visualization  tasks  with  spatial  components  but  Datey  came  to  the  conclusion  that  there  was  no 
statistical  difference  in  the  two  methods.  In  a  similar  case,  Pausch  et  al.  (1997)  predicted  that  an 
HMD  with  head  tracking  would  be  better  for  locating  items  than  a  stationary  HMD  with  hand 
controls.  However,  there  was  no  statistical  advantage  either  way.  These  tests  are  important  for 
companies  potentially  investing  in  developing  innovative  software  or  products.  The  chance  of 
losing  money  selling  a  product  not  any  more  effective  than  the  current  technology  could  be  very 
high. 

While  these  methods  compare  different  tools  to  detennine  which  one  is  the  effective 
option,  other  methods  test  to  determine  if  their  virtual  reality  tools  can  be  used  for  training.  One 
of  the  most  common  types  of  tests  is  evacuation  simulation.  As  stated  before,  virtual  simulations 
can  provide  situations  which  would  be  impossible  to  simulate  in  real  life.  For  instance,  there  are 
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health  risks  involved  in  putting  people  in  an  evacuation  setting  with  real  fire.  At  the  same  time,  a 
trainee  might  not  feel  a  sense  of  urgency  if  their  training  is  just  walking  around  in  a  safe 
building.  Orr  et  al.  (2008)  created  a  simulation  for  evacuating  an  underground  mine  fire  as  a 
response  to  this  issue.  Thirty-two  people  in  groups  of  four  went  through  the  training  simulation. 
Half  of  the  groups  were  given  a  sample  route  to  examine  first  as  a  way  to  introduce  them  to  the 
virtual  environment.  They  were  then  given  a  tougher  scenario  to  go  through.  The  other  group 
was  given  the  tougher  route  first.  Those  that  were  given  the  easiest  route  first  performed  37% 
faster  than  the  other  group.  With  this,  it  was  concluded  that  the  virtual  training  aided  the  trainees 
in  completing  the  route.  While  there  were  issues  about  whether  or  not  the  trainees  were  thinking 
for  themselves  (since  they  worked  in  groups  and  one  could  easily  just  follow  the  leader),  a 
majority  of  the  trainees  reported  that  they  felt  the  training  was  effective. 

In  a  separate  evacuation  training  program,  Molka-Danielson  and  Chabada  (2010)  created 
a  replica  of  the  first  floor  of  a  university  college  building  in  Second  Life  to  use  for  an  evacuation 
simulation.  Their  hopes  were  that  this  simulation  “could  contribute  to  an  evacuation  plan  for  the 
college  and  to  more  effective  evacuation  training  exercises  by  raising  interest.”  The  simulation 
included  putting  an  individual  into  a  burning  building  and  timing  their  ability  to  escape  the 
building  through  a  predetennined  exit.  Tables  2.2  and  2.3  show  a  pre-simulation  survey  taken  by 
the  participants  and  the  results  of  the  simulation  respectively. 
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Table  2.2:  Pre-Trial  Survey 


Pre-Trial  Surrey 

Responses 

Gender  M:F 

64V36  % 

Age  1S-25;  26-35;  35 

54*.:  36*.;  10% 

Awaie  of  safety  procedures 
in  then  workplace  or 
university? 

67%  Yes:  33%  No 

Have  you  ever  been  trained 
for  safety’  procedures 
before’’ 

72%  Yes:  2  S%  No 

How  many  people  are  in 
your  workplace  or 
university  building?  -by 
number  of  employees 

Respondents  were: 

67%  large:  (>=250) 

21%  medium:  (51-250) 
9%  small:(l  1-50) 

3%  micro;  (  =10) 

Do  you  know  where  an 
emergency  exit  at  your 
workplace  or  university  is? 

95%  Yes:  5%  No 

How  regularly  do  you  have 
safety  procedures  training? 

33%  biannuallv; 

46%  annually 

3%  once  in  6  months 

18%  never 

Have  you  expenenced  a 
fire  alarm  m  the  building? 

44%  Yes:  56%  No 

Do  you  feel  prepared  for  an 
emer  gency  situation? 

56%  Yes;  44%  No 

If  not.  what  would  make 
you  more  confident? 

32%  Practical  training 

5%  Video  training 

27%  Real  experience 

36%  Yutual  training 

Do  you  thmk  that 
computers  and  virtual 
worlds  could  be  an  useful 
simulation  tool  for  such 
emergency  training? 

64%  Yes 

6%  No 

30%  I  don't  know 

Have  you  ever  heard  of  the 
virtual  world  of  Second 

Life? 

15%Yes  have  used  SL 
45%  Yes  but  have  never 
used  SL 

40%  No; 

19 
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Table  2.3:  Trial  Results 


Nr. 

Factor 

1: 

Factor  2: 

Trial  1 
(time  in 
seconds) 

Trial  2  (time 
m  seconds) 

1 

KB 

Use 

37.75 

80.00 

“) 

KB 

Use 

23.80 

59.32 

3 

KB 

Use 

24.70 

30.70 

4 

KB 

Use 

20. SI 

43.33 

5 

KB 

Use 

30.05 

71.20 

6 

KB 

Use 

r.ss 

28.48 

7 

KB 

N  SL 

25.02 

81.19 

s 

KB 

N'SL 

37.02 

77.37 

9 

KB 

N'SL 

3S.72 

112.38 

10 

KB 

N'SL 

85.85 

77.87 

11 

Not 

N/SL 

18.06 

41.20 

12 

Not 

N'SL 

57.50 

113.2 

13 

Not 

N/SL 

220.00 

101.69 

14 

Not 

N/SL 

28.04 

159.32 

15 

Not 

N/SL 

30.77 

160.00 

16 

Not 

NSL 

27.0S 

53.20 

17 

Not 

N/SL 

32.16 

77.03 

IS 

Not 

NSL 

37.42 

N.A. 

19 

Not 

NSL 

31.37 

137.05 

20 

Not 

NSL 

36.3 

80.47 

Average  of  those  that 
know  the  building 

34.16 

66.1S 

Average  of  those  that  do 
not  know  the  building 

51.87 

102.57 

Average  for  those  already 
using  SL 

25.83 

52.17 

Average  for  those  new  to 

SL 

50.37 

97.84 

Table  2.2  provides  a  view  of  the  results  of  the  pre-trial  survey.  It  provides  an  example  of  how  to 
report  the  sample  data  and  provides  a  model  for  a  questionnaire  prior  to  using  a  simulation.  Prior 


to  the  administration  of  the  training,  the  trainees  were  split  on  virtual  training  and  more  common 


practical  training,  but  a  majority  felt  computers  would  be  helpful.  Table  2.3  shows  the  effect  of 


virtual  training.  ‘KB’  means  the  user  knew  the  building  whereas  ‘Not’  means  they  were 


unfamiliar  with  it.  ‘Use’  means  the  user  was  familiar  with  Second  Life  and  ‘N/SL’  means  the 


user  was  not  familiar  with  Second  Life.  Trial  1  and  2  represent  different  routes  with  the  second 
one  being  more  difficult.  The  times  recorded  for  those  who  were  trained  with  Second  Life 
showed  they  were  able  to  complete  both  escape  routes  in  virtually  half  the  time  as  those  who 
went  without  it.  This  type  of  comparison  is  useful  for  determining  if  virtual  training  can  be 
useful.  From  this  simulation,  it  is  clear  that  virtual  reality  had  a  significantly  positive  effect. 
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While  the  previous  methods  use  statistical  comparisons  to  confirm  the  effectiveness  of  a 
certain  technology,  others  use  a  more  subjective  approach.  Instead  of  comparing,  they  use 
questionnaires.  These  questionnaires  are  given  after  a  user  practices  with  a  technology.  The  user 
answers  the  questions  which  usually  are  along  the  lines  of,  “To  what  level  did  you  feel  a  sense  of 
presence?”  The  tester  than  analyzes  all  the  users  according  to  their  responses  and  draws 
conclusions  on  them.  One  of  the  most  common  questionnaires  is  the  University  College  London 
(UCL)  questionnaire  (Slater  et  ah,  1994;  Usoh  et  ah,  1999).  This  questionnaire  contains  seven 
questions  measuring  presence,  three  measuring  behavioral  presence,  and  three  measuring  ease  of 
locomotion.  Cheng  et  al.  (2010)  use  a  questionnaire  to  analyze  the  effectiveness  of  their  virtual 
car  engine  assembly  training.  They  used  thirty  trainees  consisting  of  twenty  employees  from 
three  different  companies  and  ten  undergraduate  engineering  students.  The  results  of  the 
questionnaire  are  shown  in  Table  2.4: 

Table  2.4:  Results  of  Engine  Assembly  Training 


Assess  Matters 

Assess  Results  (Sums  of  Trainees) 

Very 

Satisfied 

Satisfied 

Common 

Not  Good 

Very  Bad 

Content  Organization 

9 

18 

3 

Training  Strategies 

12 

15 

3 

Training  Targets 

3 

16 

9 

2 

The  Overall  Effect 

6 

21 

3 

Table  2.4  shows  how  a  strong  majority  of  the  sample  felt  the  virtual  training  was  a  very  good 
method.  In  a  similar  test,  Bajka  et  al.  (2008)  used  a  questionnaire  to  evaluate  their  virtual 
simulation  of  a  hysteroscopy.  They  took  sixty-two  gynecologists  and  put  them  all  through  twenty 
minutes  of  hands  on  virtual  training.  The  sample  of  surgeons  can  be  summarized  in  Table  2.5. 

Table  2.5:  Sample  of  Gynecologists 


Number  of  Surgeries 

Number  of  Surgeons 

>50 

26 

<50 

36 

21 
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These  participants  were  asked  to  rate  on  a  7-point  Likert  scale,  how  effective  they  felt  the 
simulation  was.  Table  2.6  shows  the  results  of  the  questionnaire  but  only  includes  the  first  three 
levels  of  the  scale,  as  no  one  rated  the  simulation  worse  than  a  “5.” 

Table  2.6:  Likert  Scale 


Absolutely 

Realistic 

Realistic 

Somewhat 

Realistic 

Scale 

7 

6 

5 

Number  of  Surgeons 

4 

40 

16 

Those  that  were  less  experienced  rated  the  training  at  an  average  of  6.48  while  the  more 
experienced  surgeons  rated  the  training  at  an  average  of  6.08.  For  this  study,  95.2%  believed  that 
the  training  was  adequate,  and  85.5%  suggested  the  training  to  be  given  to  all  inexperienced 
surgeons. 

2.5  Physiology 

While  the  previous  authors  claim  their  tests  give  credibility  to  the  effectiveness  of  virtual 
reality,  they  mostly  fail  to  cover  the  idea  of  physiological  factors.  The  test  results  are  from 
subjective  questionnaires  asking  questions  like,  “Did  you  feel  a  sense  of  presence?”  Even  if  the 
trainee  answers  to  the  highest  level  possible  depending  on  the  scale,  there  is  no  real  proof  if  they 
truly  felt  immersed  in  the  virtual  environment.  A  true  test  on  physiological  response  is  still 
missing. 

While  testing  has  been  scarce,  the  ability  to  test  for  physiological  responses  is  well  within 
the  realm  of  possibility.  Slater  et  al.  (2010)  say  that  when  a  trainee  is  going  through  a  virtual 
training  environment,  they  should  have  physiological  responses  and  should  show  behavior  that 
supports  the  idea  that  they  are  immersed  in  their  environment.  The  experimenter  just  needs  to 
know  what  to  test.  Stress  tends  to  be  the  most  common  sensation  tested.  Durrani  and  Geiger 
(2008)  claim  that  the  introduction  of  stress  into  an  experiment  causes  an  increase  in  engagement 
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and  subsequently,  in  training  effectiveness.  If  that  is  true,  physiological  factors  correlating  to 
stress  can  be  tested.  Slater  et  al.  (2009)  states  three  factors  which  correlate  to  stress  are  skin 
conductance,  heart-rate,  and  heart-rate  variability.  Meachen  et  al.  (2002)  used  these  factors,  sans 
heart-rate  variability,  to  study  stress  using  a  visual  cliff.  In  something  that  can  be  used  in  the 
future,  Durrani  and  Geiger  (2008)  propose  a  statistical  approach  to  study  knowledge  transfer  and 
skill  development  by  comparing  virtual  training  and  traditional  training  based  on  physical  cue 
fidelity  and  neurophysiologic  response. 

2.6  Future  Projects 

Durrani  and  Geiger’s  approach  is  simply  a  proposal.  There  are  a  few  more  projects 
underway  which  relate  to  virtual  trainings  and  presence.  For  instance,  Monahan  et  al.  (2009) 
want  to  develop  training  environments  for  emergency  preparedness  training  exercises.  According 
to  them,  there  are  two  alternatives  to  virtual  training:  live  exercises  and  tabletop  exercises.  Live 
exercises  saw  an  increase  in  use  for  emergency  training  following  the  September  1 1th  attacks  on 
the  World  Trade  Center  and  the  Pentagon.  Unfortunately  they  face  the  drawbacks  of  being  costly 
and  needing  a  large  number  of  volunteers  to  help  train  a  small  number  of  people.  Tabletop 
exercises  suffer  from  lack  of  visual  stimulation.  Virtual  worlds  do  not  suffer  from  these 
drawbacks  as  the  cost  of  transportation  and  loss  of  work  days  can  be  solved  by  perfonning 
trainings  in  a  virtual  world  in  which  a  person  can  connect  to  wherever  they  are  located. 
Additionally,  virtual  worlds  offer  a  visual  stimulus  not  present  in  tabletop  exercises.  Therefore, 
Monahan  et  al.  believe  the  use  of  virtual  worlds  for  emergency  training  is  more  cost  efficient  and 
more  stimulating  than  the  possible  alternatives. 

Another  future  endevor  is  being  conducted  by  Mott  and  Rajaei  (2010).  Their  goal  is  to 
design  a  system  which  uses  a  standard  webcam  and  advanced  computer  vision  techniques  to 


46 


allow  the  learners  use  of  their  hands  when  interacting  with  the  virtual  objects.  This  approach  is  a 
response  to  the  use  of  virtual  gloves  which  are  used  for  the  same  effect.  However,  they  feel  this 
is  a  less  expensive  alternative.  Cai  (2008)  wants  to  create  a  3D  environment  that  simulates  the 
real  business  world  while  at  the  same  time  provide  interfaces  to  3D  application  users  for  them  to 
interact  with  the  virtual  world.  These  future  projects  should  add  to  the  available  literature  on 
virtual  reality.  Even  without  proper  testing  procedures,  the  literature  can  still  provide  useful 
infonnation. 

2.7  Conclusion 

Because  virtual  reality  is  a  fairly  recent  innovation,  there  has  not  been  a  large  amount  of 
research  done  on  the  topic,  as  it  relates  to  learning  effectiveness.  While  there  have  been  a  fair 
share  of  simulations  using  virtual  reality,  they  have  lacked  an  adequate  amount  of  testing  to  give 
credibility  to  their  effectiveness.  Creators  of  devices,  trainings,  or  simulators  using  virtual  reality 
tend  to  confirm  the  effectiveness  of  their  creations  with  assumptions  or  subjective 
questionnaires.  Objective  evidence  of  the  effectiveness  of  virtual  reality  is  still  lacking.  A 
mixture  of  test  data  and  physiological  data  can  go  a  long  way  in  giving  credibility  to  virtual 
reality  as  a  testing  method.  Bronak  at  al.  (2006)  state,  “We  believe  that  virtual  worlds  support 
deep  learning  and  can  help  learners  make  meaning  in  ways  similar  to  outside... environments. 
Our  experience  suggests  that  virtual  worlds  offer  participants  a  sense  of  presence,  immediacy, 
movement,  artifacts,  and  communication  unavailable  within  traditional  Internet-based  learning 
environments.”  Now  the  only  thing  to  do  is  attain  data  that  can  give  credibility  to  this  assertion. 
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CHAPTER  III 

METHODOLOGY 

3.1  Experiment  Overview 

In  order  to  verify  if  virtual  worlds  can  be  as  effective  as  more  traditional  methods,  a 
comparative  study  is  created  in  which  two  instruction  types,  Second  Life  and  a  traditional  set  of 
paper-and-pen  exercises  called  case  studies,  are  used.  The  subjects  in  the  study  are  evenly 
separated  into  the  two  groups  and  then  tested  on  the  final  day  of  the  experiment.  The  results  from 
this  final  assessment  are  used  to  determine  if  one  of  the  instruction  types  is  more  effective  than 
the  other.  The  procedure  for  this  experiment  is  outlined  in  Figure  3.1. 


Figure  3.1:  Block  Diagram  of  Experiment 

On  Day  1,  the  subjects  go  through  a  preliminary  training  where  they  learn  about 
improvised  explosive  devices  (IED’s).  The  subjects  leam  how  IED’s  work  and  the  damage  they 
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can  cause.  In  addition  they  learn  the  procedures  for  defusing  a  bomb.  In  Day  2  the  subjects  are 
separated  into  two  groups:  a  paper  and  pen  training  group  using  classical  case  studies  and  a 
virtual  environment  training  group  using  the  Second  Life  software.  In  this  part  of  the  training, 
the  subjects  apply  what  they  learned  in  Day  1  to  simulated  situations  and  are  provided  with 
feedback  in  order  to  gain  experience.  The  difference  is  in  the  medium  in  which  they  experience 
the  simulations.  On  the  final  day,  the  subjects,  one  at  a  time,  apply  what  they  have  learned  in  a 
live  scenario  with  a  victim  played  by  an  actor  or  actress.  They  are  given  a  time  limit  and  are 
graded  on  how  well  they  perfonn  the  necessary  procedures. 

In  addition  to  the  training,  each  subject  fills  out  a  series  of  questionnaires  throughout  the 
course  of  the  experiment.  The  numbers  on  the  bottom  of  the  blocks  in  Figure  3.1  refer  to  the 
questionnaires  completed  during  the  experiment.  The  placement  of  the  numbers  (left  and  right) 
indicates  when  in  the  day  the  subjects  take  the  questionnaires  (before  and  after  the  training 
respectively).  The  questionnaires  are  listed  below  in  Table  3.1. 


Table  3.1:  List  of  Questionnaires 


# 

Questionnaire 

1 

Screening  Form 

2 

Consent  Form 

3 

Demographics 

4 

The  Big  Five  Inventory 

5 

Sensation  Seeking  Scale 

6a 

Case  Studies  Questionnaire 

6b 

Second  Life  Questionnaire 

7 

Bomb  Training  Evaluation  Form 

8 

State-Trait  Anxiety  Test 

9 

Edinburgh  Handedness  Inventory 

10 

Post  Experiment  Questions  -  Anxiety  Questionnaire 

The  details  of  the  experiment  are  organized  by  the  day  in  which  the  activities  occur.  First  is  the 
preliminary  training  in  Day  1 . 
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3.2  Day  1:  Preliminary  Training 

At  the  start  of  the  experiment  the  subjects  are  introduced  to  the  experiment’s  conductors 
and  are  told  about  how  the  payment  for  their  time  is  administered.  The  subjects  are  also  warned 
not  to  try  to  utilize  this  training  in  a  real  situation.  The  material  presented  to  them  is  for  research 
purposes  only,  and  it  lacks  enough  depth  to  fully  qualify  the  subjects  as  bomb  experts.  After  the 
initial  introduction,  each  subject  is  required  to  complete  the  Screening  Form.  This  form  is  used  to 
screen  out  any  subjects  that  would  have  a  high  chance  of  being  harmed  by  the  content  of  the 
experiment  or  subjects  that  do  not  meet  the  age  or  school  registration  requirements.  After  the 
screening  is  completed,  the  subjects  are  given  the  Consent  Fonn  in  which  they  grant  pennission 
to  be  used  as  human  subjects.  The  nature  of  the  experiment  is  described  and  the  subjects  are  told 
they  may  opt  out  of  the  experiment  whenever  they  wish.  After  signing  the  Consent  Form  the 
subjects  are  given  note  cards  with  their  Subject  Numbers  and  their  log-in  information.  The 
numbers  are  provided  randomly  and  range  from  1  to  65.  With  their  note  card,  each  subject  logs 
into  the  experiment’s  web  page  and  begins  their  preliminary  training  by  completing  additional 
questionnaires. 

The  three  questionnaires  completed  by  the  subjects  prior  to  beginning  their  training  are 
the  Demographics  Questionnaire,  The  Big  Five  Inventory,  and  the  Sensation  Seeking  Scale.  The 
Demographics  Questionnaire  asks  questions  about  their  history  with  Second  Life  along  with 
some  general  questions  about  age  and  college  grade  level.  The  Big  Five  Inventory  is  a 
personality  test  that  rates  subjects  based  on  five  factors:  openness,  conscientiousness, 
extraversion,  agreeableness,  and  neuroticism.  The  Sensation  Seeking  Scale  is  another  personality 
test.  It  is  used  to  assess  the  subjects’  tendency  to  engage  in  spontaneous  and  potentially 
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dangerous  behavior.  After  taking  the  questionnaires,  the  subjects  can  move  on  to  the  actual 
training. 

3.2.1  Preliminary  IED  Training 

To  begin  the  training,  the  subjects  read  through  a  file  that  contains  seven  links.  The  first 
link  contains  some  basics  about  explosives.  The  second  gives  information  about  how  bombs 
cause  damage.  The  third  provides  the  subject  with  an  option  to  open  a  file  containing  information 
on  render  safe  procedures  as  either  a  Word  document  or  a  PDF  file.  The  fourth  link  opens  a  PDF 
document  where  the  render  safe  procedures  are  detailed.  After  reading  the  PDF  document,  the 
subjects  open  a  Soft  Chalk  file  in  the  fifth  link  which  has  more  information  regarding  the  effects 
of  IED  explosions  and  the  process  that  initiates  the  explosive.  Using  this  knowledge,  the  subject 
leams  how  to  defuse  an  IED  by  learning  the  components  of  the  bomb  and  the  order  to  defuse  it 
without  accidently  causing  it  to  explode.  The  subject  is  also  given  several  matching  exercises  to 
practice  learning  the  components  of  IED’ s  and  the  procedures  for  defusing  explosives. 
Additionally,  the  subjects  are  given  note  cards  with  the  render  safe  procedures  typed  out  in  order 
to  help  them  remember  the  procedures.  After  they  have  completed  the  training,  the  subjects  take 
an  assessment  test  in  the  sixth  link  to  assess  how  well  they  learned  the  material. 

3.3  Day  2:  Case  Studies  or  Second  Life  Training 

On  the  second  day,  the  subjects  are  split  into  two  groups:  a  paper  and  pen  training  group 
(case  studies)  and  a  virtual  environment  training  group  (Second  Life  software).  The  subjects  in 
the  paper  and  pen  training  group  are  the  ones  that  received  an  odd  number  for  their  subject 
number.  The  even-numbered  subjects  take  the  virtual  environment  training.  The  two  types  of 
training  are  administered  separately.  Therefore  they  are  discussed  in  individual  sections  aptly 
labeled  Case  Studies  and  Second  Life  respectively. 
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3.3.1  Case  Studies 

Before  starting  the  case  studies,  the  subjects  are  given  a  chart  which  helps  them  evaluate 
the  blast  radius  of  an  IED  and  the  corresponding  distance  that  is  safe  for  evacuation.  The  subjects 
are  presented  with  three  case  studies.  In  each  case,  the  subject  is  provided  with  information  about 
the  situation.  The  subject  completes  a  series  of  matching  exercises  and  multiple  choice  questions. 
After  answering  the  questions,  the  subject  assesses  how  the  explosive  looks  and  proceeds  to 
draw  the  explosive  and  label  its  components.  The  subject  completes  the  case  by  completing  an 
ordering  exercise  in  which  they  label  each  wire  of  the  explosive  in  the  order  it  is  to  be  cut.  After 
each  case  study  is  completed,  the  subjects  are  provided  with  the  answers  to  the  questions  and 
then  the  next  case  begins.  The  three  cases  are  described  below: 

3.3. 1.1  Case  Study  1:  Unvacated  Business  Building 

Early  Thursday  morning,  a  security  officer  of  BioPharm  Corp.  finds  a  bomb  on  the  first 
floor  stairway  while  doing  a  routine  security  walk  through  of  the  30  story  building.  The  building 
is  located  in  the  center  of  downtown  Dallas,  TX.  Upon  discovering  the  explosive,  the  officer 
contacted  911  for  help.  This  is  where  the  subject  comes  into  the  case  study.  When  they  arrive, 
the  subject  is  met  by  the  security  officer  who  provides  them  further  details. 

Details 

Bomb  Location:  Hidden  Behind  first  floor  Stairway 

Civilians:  The  building  cannot  be  totally  evacuated.  There  are  a  number  of  labs  that 

require  personnel  to  remain  in  order  to  protect  the  community. 

Possible  Suspects:  Police  reinforcements  are  patrolling  the  surrounding  area  and 

reviewing  security  tapes,  but  have  not  found  any  suspects. 
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Bomb  Visual :  The  case  study  is  accompanied  with  pictures  of  the  bomb  shown  in  Figure 
3.2. 


Figure  3.2:  Bomb  in  Case  Study  1 


3.3. 1.2  Case  Study  2:  Vacated  Bus  on  Active  Highway 

A  60  passenger  Greyhound  bus  which  left  from  Houston,  Texas  en  route  to  the  McAllen 
Bus  Station  has  stopped  on  the  side  of  Highway  281  near  by  Monte  Cristo.  One  of  the  passengers 
in  the  bus,  having  moved  to  a  recently  vacated  seat,  discovered  a  box  with  a  bomb  inside.  When 
you  arrive  you  are  greeted  by  a  police  man  who  notifies  you  that  there  are  current  efforts  on  the 
way  to  block  the  highway. 

Details 

Bomb  Location :  Found  on  top  of  3ld  row  seat 

Causality  Risks:  Bus  emptied,  however  cars  are  still  occupying  the  road  by  the  busses 

current  location. 

Possible  Suspects:  Passenger  previously  occupying  the  seat  where  the  bomb  is  located 

exited  on  the  previous  stop.  All  efforts  are  being  made  to  locate  suspect. 
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Bomb  Visual :  The  case  study  is  accompanied  with  pictures  of  the  bomb  shown  in  Figure 
3.3. 


Figure  3.3:  Bomb  in  Case  Study  2 


3.3. 1.3  Case  Study  3:  School  Janitor’s  Room 

In  the  afternoon  of  April  5th,  2009,  the  Janitor  of  Edinburg  Junior  High  went  to  the 
Janitor’s  room  to  retrieve  a  mop  in  order  to  clean  up  a  spill  that  took  place  in  the  cafeteria.  When 
reaching  for  the  mop,  the  Janitor  noticed  a  foreign  object  located  on  the  floor  of  the  room.  Upon 
closer  inspection  he  realized  it  was  a  bomb  and  quickly  ran  to  the  Administration  office  to  notify 
the  principal.  The  principal  then  notified  authorities,  which  luckily  had  located  you  there  for  your 
son’s  Career  Day.  You  are  quietly  asked  to  step  out  of  the  room  and  then  notified  of  the  situation 
by  the  principal. 

Details: 

Bomb  Location :  On  1st  floor  Janitor  Room,  placed  on  the  floor. 

Causality  Risks :  The  school  has  barely  begun  the  evacuating  process  of  the  Jr.  High 
School.  You  are  told  that  on  fire  drills  it  takes  approximately  10  minutes  for  school  to  be 
evacuated. 
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Additional  Information :  Janitor  had  noticed  that  the  tinier  seemed  to  stay  on  15  minutes 
and  49  seconds.  But  unsure  if  that  is  a  decoy.  There  is  also  a  woman  tied  up  near  the 
bomb. 

Suspects :  No  known  suspects. 

Bomb  Visual :  The  case  study  is  accompanied  with  pictures  of  the  bomb  shown  in  Figure 
3.4. 


Figure  3.4:  Bomb  in  Case  Study  3 

3.3. 1.4  Questionnaires 

Following  the  completion  of  the  three  case  studies,  the  subjects  complete  two 
questionnaires.  The  first  is  the  Case  Study  Questionnaire  which  assesses  the  effectiveness  of  the 
Case  Studies  in  properly  training  the  subjects.  The  second  questionnaire  is  the  Bomb  Training 
Evaluation  Fonn  located  in  the  seventh  link  of  the  file  from  the  first  day  of  training.  This  fonn 
assesses  the  effectiveness  of  the  entire  training  according  to  the  subjects.  After  they  complete 
their  questionnaires,  the  subjects  are  now  ready  to  take  their  final  test. 
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3.3.2  Second  Life 

Whereas  the  odd-numbered  subjects  participated  in  the  case  studies,  the  even-numbered 
subjects  participate  in  the  Second  Life  training.  The  Second  Life  training  consists  of  the  same 
scenarios  used  in  the  case  studies  and  in  the  same  order.  However,  the  beginning  of  the  training 
begins  with  a  tutorial  on  how  to  use  some  basic  functions  of  Second  Life.  Using  the  log-in 
information  on  their  cards,  the  subjects  enter  the  Second  Life  virtual  world.  The  subjects  start  at 
the  outside  of  a  training  facility  where  they  first  leam  how  to  walk.  A  screen  shot  of  this  tutorial 
is  shown  in  Figure  3.5. 


Figure  3.5:  Second  Life  Walking  Tutorial 

After  walking  through  the  door,  the  subjects  leam  how  to  touch  objects  and  zoom  in  and  out  to 
get  a  better  view  of  them.  Figure  3.6  and  Figure  3.7  show  this  part  of  the  tutorial. 
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Figure  3.6:  Second  Life  Touching  Objects  Tutorial 


Figure  3.7:  Second  Life  Zoom  Tutorial 

Using  the  zoom  feature,  the  subjects  can  look  at  an  IED  and  practice  defusing  it  by  clicking  on 
the  wires.  After  practicing  with  the  IED,  the  subjects  look  at  sign  with  instructions  for 
progressing  through  the  scenarios.  A  screenshot  of  this  instructional  sign  is  shown  in  Figure  3.8. 
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Figure  3.8:  Second  Life  Instructions 

The  instructions  on  the  sign  tell  the  subjects  to  first  click  on  the  police  officer  to  gather 
information  on  the  scenario.  To  practice  this,  a  police  officer  is  placed  near  the  sign.  If  the 
subjects  click  on  him,  they  are  shown  a  window  containing  different  types  of  information  that  the 
subjects  can  gather  from  the  officer.  This  is  shown  in  Figure  3.9.  Next  the  subject  answers  the 
same  multiple  choice  questions  provided  to  the  Case  Study  group.  While  a  programmed 
‘Question  Bot’  was  to  be  used  for  the  questions,  multiple  problems  led  to  the  questions  being 
typed  on  a  Word  document  and  displayed  to  the  subject.  After  answering  the  multiple  choice 
questions,  the  subject  defuses  the  IED.  After  defusing  the  IED,  the  subject  reviews  the  scenario 
before  moving  on  to  the  next  scenario.  The  review  includes  the  proper  evacuation  distance,  the 
threat  level,  the  steps  to  ensure  public  safety,  and  the  procedure  to  defuse  the  IED. 
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Figure  3.9:  Second  Life  Police  Officer  Tutorial 

After  completing  the  tutorial,  the  subjects  continue  to  the  scenarios  which  are  identical  to  those 
in  the  Case  Studies. 

3.3.2. 1  Questionnaires 

Following  the  completion  of  the  Second  Life  training,  each  of  the  subjects  complete  two 
questionnaires.  The  first  is  the  Second  Life  Questionnaire  which  assesses  the  effectiveness  of 
Second  Life  in  properly  training  the  subjects.  The  second  questionnaire  is  the  Bomb  Training 
Evaluation  Form.  This  form  assesses  the  effectiveness  of  the  entire  training  according  to  the 
subjects.  After  the  subjects  complete  their  questionnaires,  they  are  ready  to  for  the  final  day  of 
the  experiment. 
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3.4  Day  3:  Final  Scenario 

3.4.1  Preliminary  Questionnaires 

The  final  day  of  the  training  begins  with  the  test  subject  taking  the  State-Trait  Anxiety 
Inventory.  In  this  inventory  the  subjects  are  given  several  statements.  The  subject  rates 
themselves  on  a  scale  of  1  to  4  with  1  being  that  the  statement  does  not  describe  them  at  all  and  4 
being  that  the  statement  describes  them  “very  much  so.”  After  taking  the  State-Trait  Anxiety 
Inventory,  the  subject  is  verbally  given  the  Edinburgh  Handedness  Inventory.  In  this 
questionnaire,  the  subject  is  asked  whether  they  use  their  right  or  left  hand  for  a  series  of 
everyday  activities  such  as  brushing  their  teeth  or  writing.  The  purpose  of  the  questionnaire  is  to 
determine  the  subject’s  dominant  hand  so  not  to  interfere  with  attachment  of  electro-dennal 
sensors.  After  completing  the  questionnaires,  the  sensors  for  the  physiological  measurement 
detailed  in  the  next  section  are  placed  on  the  subject. 

3.4.2  Physiological  Arrangement 

The  physiological  equipment  being  used  in  this  experiment  is  provided  by  BIOP  AC 
Systems,  Inc.  The  equipment  consists  of  biotelemetry  modules  designed  to  receive  signals  from 
battery  charged  transmitters.  The  transmitters  that  are  connected  to  the  physiological  transducers 
relay  the  signals  wires  that  are  either  clipped  or  attached  with  an  adhesive  to  the  subject.  The 
physiological  measurement  being  taken  is  the  subject’s  heart  rate  using  an  Electrocardiogram 
(ECG). 

3.4.2. 1  ECG 

Figure  3.10  displays  electrode  placement  for  the  ECG.  First  the  skin  is  prepared  by  using 
an  alcohol  swab  to  clean  off  any  excess  dirt  or  oil  from  the  skin.  Next  three  pads  are  attached  to 
the  subject’s  chest:  one  under  each  pectoral  muscle,  and  another  underneath  the  right  clavicle.  A 
red,  white,  and  black  clip  lead  is  used  to  set  up  the  ECG.  The  red  wire  is  clipped  under  the  left 
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pectoral  muscle  and  the  black  wire  is  clipped  under  the  right  pectoral  muscle.  Finally,  the  white 
wire  is  clipped  under  the  right  collarbone.  The  three-wire  lead  is  attached  to  an  ECG  transmitter 
and  the  transmitter  is  taped  to  the  subject’s  shirt. 


Figure  3.10:  ECG  Arrangement 


3.4.3  Final  Test 

To  begin  the  test,  the  subject  sits  still  for  ten  minutes  in  order  to  get  a  preliminary 
baseline  of  their  physiological  measurements  and  to  acclimate  to  the  situation.  After  the  ten 
minutes  are  completed,  one  of  the  researchers  enters  the  room  and  presents  the  scenario  to  the 
subject.  The  subject  reads  the  scenario  and  is  given  a  notepad,  a  pen,  and  a  flashlight  as  their 
resources.  When  they  are  ready,  they  are  taken  into  the  scenario  room  with  a  victim  who  has  a 
mock  bomb  attached  to  them.  The  subject  has  15  minutes  to  defuse  the  bomb.  While  the  subject 
is  defusing  the  bomb,  they  are  graded  on  how  they  perfonn  the  actions  they  were  trained  to 
perform.  After  the  scenario  is  completed,  the  subject  has  the  sensors  removed  and  is  taken 
outside  to  take  one  final  questionnaire.  This  questionnaire  assesses  the  anxiety  the  subject  says 
they  felt  during  the  scenario. 

The  completion  of  the  final  questionnaire  marks  the  conclusion  of  the  experiment. 
Following  the  conclusion  of  the  experiment,  all  the  data  was  collected  and  organized  for 
analysis.  In  the  next  chapter,  the  data  from  this  experiment  is  analyzed  in  order  to  draw 
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inferences  about  whether  one  instruction  type  was  more  effective  than  the  other  at  teaching  the 
subjects  how  to  defuse  an  explosive  device. 
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CHAPTER  IV 

DATA  ANALYSIS 

4.1  Outline  of  Data  Analysis 

As  explained  in  the  previous  chapter,  this  experiment  is  performed  over  a  three  day  span. 
Therefore,  the  data  gathered  from  this  experiment  is  presented  in  the  order  in  which  it  is 
collected.  An  overview  of  the  subjects’  demographic  information  is  provided  first  as  are  the 
results  of  the  assessment  the  subjects  complete  following  the  conclusion  of  their  first  day  of 
training.  The  results  of  the  Big  Five  Inventory  and  the  Sensation  Seeking  Scale  are  provided  in 
Appendix  1  and  2  respectively. 

On  the  second  day  of  their  training,  the  subjects  are  broken  up  into  two  groups  and 
receive  a  specialized  instruction  based  on  their  grouping.  The  even-numbered  subjects  received 
the  Second  Life  instruction  while  the  odd-numbered  subjects  received  the  case  studies 
instruction.  Following  the  completion  of  their  training,  each  subject  completes  a  questionnaire 
assessing  their  opinion  of  the  effectiveness  of  their  particular  training.  The  results  of  the 
questionnaires  are  provided  along  with  the  result  of  the  Bomb  Training  Evaluation  Form,  a 
questionnaire  assessing  the  effectiveness  of  the  training  overall. 

On  the  final  day,  each  subject  completes  their  final  assessment.  Prior  to  completing  the 
assessment,  the  subjects  each  take  the  State-Trait  Anxiety  Questionnaire,  a  questionnaire 
designed  to  assess  the  difference  between  a  subjects’  nonnal  anxiety  and  the  anxiety  the  current 
situation  is  eliciting.  The  results  of  this  questionnaire  are  provided  in  Appendix  3.  The 
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purpose  of  this  experiment  is  to  determine  if  the  instruction  type  has  an  effect  on  the  subjects’ 
ability  to  learn  and  perform  a  task.  The  result  of  testing  whether  the  instruction  type  affected  the 
subjects’  performance  in  terms  of  completing  render  safe  procedures  and  correctly  defusing  the 
bomb  is  provided.  In  addition,  a  test  is  conducted  to  assess  whether  subjects  that  took  one 
instruction  type  were  calmer  than  the  subjects  that  took  the  other.  This  is  done  by  using 
physiological  measurements  recorded  during  the  final  day  of  the  experiment.  For  this 
experiment,  the  measurements  analyzed  are  the  subjects’  heart  rates.  Finally  an  overview  of  the 
subjects’  perception  of  the  final  assessment  is  provided. 

4.2  Day  1 

4.2.1  Demographics 

This  experiment  was  limited  to  current  male  students  of  the  University  of  Texas-Pan 
American.  In  addition,  the  students  had  to  be  eligible  to  work  in  the  United  States.  Potential 
subjects  that  noted  a  history  of  epilepsy,  seizures,  post-traumatic  stress  disorder,  or  panic  attacks 
were  disqualified  from  the  experiment.  Sixty-five  male  participants  participated  in  this 
experiment.  On  the  first  day  of  their  training,  the  subjects  were  given  a  survey  asking  for 
demographic  information  as  well  as  their  familiarity  with  Second  Life.  The  questions  asked  are 
as  follows: 

Question  1 :  What  is  your  age? 

Question  2:  What  is  your  classification? 

Question  3:  Please  indicate  your  current  level  of  familiarity  with  computer  games  and 

gaming. 

Question  4:  Please  indicate  your  current  level  of  awareness  about  Second  Life,  on  a  scale 


of  1  to  6. 
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Question  5:  Approximately  how  many  months  ago  did  you  first  use  Second  Life?  (Please 
enter  only  whole  numbers.  For  example,  if  you  started  using  it  3  years  ago,  enter  36;  if 
you  have  never  used  it  before,  enter  0): _ months  ago. 

Question  6:  Please  indicate  how  frequently  you  use  Second  Life  for  personal  leisure  or 
communication,  on  a  scale  of  1  to  6: 

The  responses  for  Question  1  are  summarized  in  Figure  4.1. 


Histogram  of  Age 


Age 

Figure  4.1:  Subjects'  Ages 

Figure  4. 1  shows  that  a  strong  majority  of  the  subjects  were  between  the  ages  of  18-20  which  is 
expected  as  those  are  typically  the  years  that  people  attend  college.  The  classification  of  each  of 
the  subjects  was  examined  next.  Figure  4.2  summarizes  the  subjects’  responses  in  regards  to  their 
college  classification. 
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Chart  of  Classification 


Classification 

Figure  4.2:  Subjects'  Classifications 

The  distribution  of  classifications  is  almost  uniform.  Seniors  and  sophomores  make  up  a  majority 
of  the  subjects  while  there  were  an  almost  equal  number  of  freshmen  and  juniors.  Figure  4.3 
displays  how  much  familiarity  with  computer  games  the  subjects  have.  They  were  asked  to 
respond  based  on  the  following  scale: 

1 :  Never  played  computer  games 

2:  Little  experience  with  computer  games 

3:  Some  experience  with  computer  games 

4:  Fair  amount  of  experience  with  computer  games 

5:  Moderate  experience  with  computer  games 

6:  Significant  experience  with  computer  games 
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Histogram  of  Familiarity  with  Computer  Games 
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FamHiarity  with  Computer  Games 

Figure  4.3:  Subjects'  Familiarity  with  Computer  Games 

According  to  Figure  4.3,  most  of  the  subjects  rated  at  a  3  or  above  with  only  three  subjects 
mentioning  that  they  had  little  to  no  computer  game  experience.  Their  awareness  with  Second 
Life  was  quite  different.  The  subjects  were  asked  to  rank  their  awareness  on  1-6  scale  with  1 
meaning  that  they  never  heard  of  Second  Life  and  6  meaning  they  are  significantly  familiar  with 
Second  Life.  Figure  4.4  shows  that  over  half  of  the  subjects  did  not  know  about  Second  Life  and 
only  one  subject  reported  they  were  significantly  familiar. 


Histogram  of  Awareness  of  SL 


Awareness  of  SL 


Figure  4.4:  Subjects'  Awareness  of  Second  Life 
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Due  to  the  majority  of  the  group  lacking  any  experience  with  Second  Life,  a  majority  of  the 
group  reported  that  they  had  no  experience  using  Second  Life  as  almost  all  of  them  recorded  a 
response  of  0.  Figure  4.5  shows  this  result. 


Histogram  of  Months  Since  Last  Use  of  SL 


Months  Since  Last  Use  of  SL 

Figure  4.5:  Months  since  Subjects'  Last  Usage  of  Second  Life 

Unsurprisingly,  as  is  shown  in  Figure  4.6,  all  but  one  of  the  subjects  recorded  a  1  for  their 
frequency  of  use  of  Second  Life  for  leisure.  Since  this  was  also  supposed  to  be  on  a  1-6  scale 
with  1  meaning  ‘I  do  not  use  it  at  all’  and  6  meaning  ‘I  use  Second  Life  very  frequently’  the 
results  indicate  that  almost  no  one  in  the  group  uses  Second  Life. 
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The  results  of  the  demographics  survey  reveal  that  a  strong  majority  of  the  subjects  would  be 
introduced  to  something  new  if  they  were  to  receive  the  Second  Life  training. 

4.2.2  Render  Safe  Assessment  Summary 

Prior  to  concluding  their  first  day  of  training,  the  subjects  complete  the  Render  Safe 
Assessment.  This  assessment  is  meant  to  test  the  subjects’  knowledge  of  what  they  read  during 
the  first  day  of  training.  A  summary  of  the  results  for  the  subjects  is  shown  in  Figure  4.7. 


Summary  for  RSP  Assessment 
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Figure  4.7:  Summary  for  Render  Safe  Assessment 
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The  results  show  that  on  average  the  subjects  obtained  a  score  of  about  61.5  out  of  a  possible  80 
points.  The  lowest  score  was  a  23.33  while  the  highest  was  the  maximum  score  of  80.  The 
confidence  interval  for  the  mean  shows  that  with  95%  confidence,  the  mean  falls  between  58.04 
and  64.98. 

4.3  Day  2 

As  previously  mentioned,  the  subjects  are  broken  up  into  their  respective  groups  on  the 
second  day  of  their  training.  Day  2  consists  of  each  group  going  through  a  specialized  training 
with  odd-numbered  subjects  receiving  the  case  studies  instruction  and  the  even-numbered 
subjects  receiving  the  Second  Life  instruction.  Following  their  training,  each  subject  completes  a 
survey  for  the  instruction  type  under  which  they  are  trained  and  then  completes  a  survey 
assessing  their  opinion  of  the  training  overall. 

4.3.1  Case  Studies  Questionnaire 

The  Case  Studies  Questionnaire  consists  of  six  questions  with  responses  recorded  on  a  6- 
point  Likert  scale  where  1  means  ‘Strongly  Disagree’  and  6  means  ‘Strongly  Agree.’  The 
questions  provided  are  as  follows: 

Question  1 : 1  experienced  a  high  level  of  interaction  in  the  case  studies  scenarios. 

Question  2:  The  case  studies  did  a  very  poor  job  of  using  a  story  to  explain  tasks. 

Question  3: 1  believed  that  I  was  a  character  in  the  scenarios. 

Question  4:  The  case  studies  were  very  unrealistic. 

Question  5:  The  writing  was  very  descriptive. 

Question  6:  The  case  studies  training  was  not  at  all  engaging. 

The  questions  alternate  between  positively  and  negatively  connoted  questions  to  avoid  subjects 
just  choosing  the  same  answer  for  every  question.  Figures  4.8  through  4.13  summarize  the 
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subjects’  responses  to  each  of  the  questions.  Note  that  instances  of  an  asterisk  denote  that  a 
subject  failed  to  answer  one  of  the  questions. 

Figure  4.8  displays  the  results  of  the  subjects’  responses  to  the  first  question. 


Chart  of  1 


l 

Figure  4.8:  Summary  of  Case  Studies  Question  1 

shows  that  a  majority  of  the  subjects  found  the  case  studies  to  be  interactive  with  a 
being  the  most  common  response.  Since  the  positively  connoted  question  was  mostly 
answered  with  high  marks  it  can  be  expected  that  a  negatively  connoted  question  would  be 
answered  with  mostly  low  marks.  Figure  4.9  displays  exactly  this  trend. 


Figure  4.8 
rating  of  5 
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Chart  of  2 


2 

Figure  4.9:  Summary  of  Case  Studies  Question  2 

According  to  Figure  4.9,  most  of  the  subjects  disagreed  strongly  about  the  assertion  that  the  case 
studies  did  a  poor  job  of  using  a  story  to  explain  the  tasks  they  were  to  complete.  Figure  4.10 
displays  the  results  of  the  third  question  which  asked  if  the  subjects  felt  they  were  a  character  in 
the  scenario.  The  results  are  positive,  but  the  highest  rated  response  is  a  neutral  ‘Somewhat 
Agree.’ 


Chart  of  3 


3 


Figure  4.10:  Summary  of  Case  Studies  Question  3 
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Figure  4.11  shows  that  the  subjects  disagreed  with  the  assertion  that  the  case  studies  were 
unrealistic  showing  that  they  felt  the  scenarios  could  take  place  in  a  real-life  environment. 


Chart  of  4 
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Figure  4.11:  Summary  of  Case  Studies  Question  4 

Figure  4.12  shows  that  the  subjects’  thoughts  on  the  descriptiveness  of  how  the  cases  were 
written  vary  widely.  While  more  subjects  agreed  with  the  assertion  than  disagreed,  there  was  still 
a  strong  contingency  that  disagreed. 


Chart  of  5 
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Figure  4.12:  Summary  of  Case  Studies  Question  5 

The  results  of  the  final  question  are  displayed  in  Figure  4.13. 
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Chart  of  6 
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Figure  4.13:  Summary  of  Case  Studies  Question  6 

The  results  indicate  that  the  subjects  found  the  case  studies  engaging  although  there  were  a 
couple  of  subjects  that  felt  strongly  agreed  with  the  assertion  that  the  case  studies  were  not  at  all 
engaging.  The  results  of  the  questionnaire  imply  that  there  was  a  general  good  feeling  towards 
the  case  studies.  In  the  next  section,  a  similar  questionnaire  assessing  the  subjects’  opinion  about 
the  Second  Life  training  is  provided. 

4.3.2  Second  Life  Questionnaire 

While  one  group  took  the  case  studies  training,  the  other  group  was  taking  the  Second 
Life  training.  After  completing  their  training,  the  subjects  completed  a  questionnaire  similar  to 
that  of  the  case  studies  group.  The  subjects  responded  to  the  questions  using  the  same  6-point 
Likert  scale  as  used  in  the  case  studies  training.  The  questions  asked  in  the  questionnaire  are 
listed  below: 

Question  1 :  I  experienced  a  high  level  of  interaction  in  the  Second  Life  scenarios. 
Question  2:  The  Second  Life  did  a  very  poor  job  of  using  a  story  to  explain  tasks. 
Question  3:  I  believed  that  I  was  a  character  in  the  scenarios. 

Question  4:  The  Second  Life  scenarios  were  very  unrealistic. 
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Question  5:  The  sound  effects  were  very  realistic. 

Question  6:  The  Second  Life  training  was  not  at  all  engaging. 

Just  as  with  the  case  studies  questionnaire,  the  questions  in  the  Second  Life  questionnaire 
alternate  between  positively  and  negatively  connoted  questions  to  avoid  subjects  just  choosing 
the  same  answer  for  every  question.  Figures  4.14  through  4.19  summarize  the  subjects’ 
responses  to  each  of  the  questions.  Again,  note  that  instances  of  an  asterisk  denote  that  a  subject 
failed  to  answer  one  of  the  questions. 


Chart  of  1 
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Figure  4.14:  Summary  of  Second  Life  Question  1 

According  the  Figure  4. 14,  a  majority  of  the  subjects  reported  that  they  felt  a  high  level 
of  interaction  while  they  were  training  in  Second  Life.  A  majority  of  the  subjects  also  reported 
that  they  felt  the  training  did  a  good  job  of  using  stories  to  explain  the  tasks  as  indicated  with  the 
subjects  disagreeing  with  the  negative  wording  of  the  question.  This  is  shown  in  Figure  4.15. 
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Chart  of  2 
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Figure  4.15:  Summary  of  Second  Life  Question  2 

The  next  question  received  mixed  responses.  About  the  same  amount  of  people  reported 
that  they  felt  they  were  the  character  in  the  scenarios  as  the  people  who  said  they  did  not  feel  this 
way.  The  results  for  Question  3  are  shown  in  Figure  4.16. 


Chart  of  3 
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Figure  4.16:  Summary  of  Second  Life  Question  3 

In  terms  of  level  of  realism,  the  subjects  were  also  mixed  in  their  responses.  As  shown  in 
Figure  4. 17,  most  of  the  responses  fall  under  the  two  most  neutral  categories  ‘Agree’  and 
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‘Disagree.’  Therefore  it  can  be  concluded  that  the  perceived  level  of  realism  was  not  very  high 
for  most  of  the  subjects. 


Chart  of  4 
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Figure  4.17:  Summary  of  Second  Life  Question  4 

In  terms  of  the  sound  effects,  the  reviews  were  also  mixed.  This  is  shown  in  Figure  4.18.  It  is  to 
no  surprise  that  the  realism  of  the  sound  effects  would  be  perceived  similarly  as  the  realism  of 
the  entire  training  as  a  whole. 


Chart  of  5 


5 


Figure  4.18:  Summary  of  Second  Life  Question  5 

The  results  for  the  final  question  are  shown  in  Figure  4.19. 


54 


77 


Chart  of  6 


Figure  4.19:  Summary  of  Second  Life  Question  6 

The  results  indicate  that  the  subjects  disagreed  with  the  assertion  that  the  Second  Life  training 
was  not  engaging.  Overall,  the  surveys  indicate  generally  positive  perceptions  of  the  Second  Life 
training  although  some  questions  tended  to  be  answered  rather  neutrally,  especially  when  it  came 
to  the  realism  of  the  training.  This  indicates  some  improvements  can  be  made  to  the  program 
which  could  improve  the  effectiveness  of  the  Second  Life  training.  The  next  section  contains  the 
results  of  the  final  survey  taken  during  the  second  day  of  training. 

4.3.3  Post  Training  Student  Survey 

The  final  survey  administered  on  the  second  day  of  training  is  the  Post  Training  Student 
Survey.  This  survey  assesses  the  perceived  quality  of  the  study  as  a  whole.  In  total  there  were  9 
questions  in  the  survey,  but  one  question  was  repeated.  Therefore,  the  responses  to  question  4 
have  been  removed.  The  questions  were  answered  using  a  6-point  Likert  scale  ranging  from  a  ‘  1  ’ 
meaning  ‘Strongly  Disagree’  to  ‘6’  meaning  ‘Strongly  Agree.’  The  questions  of  the  survey  are  as 
follows: 

1 .  The  training  met  my  expectations. 

2.  I  will  be  able  to  apply  the  knowledge  learned. 
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3.  The  content  was  organized  and  easy  to  follow. 

5.  The  materials  distributed  were  pertinent  and  useful. 

6.  The  trainer  was  knowledgeable. 

7.  The  quality  of  instruction  was  good. 

8.  The  trainer  met  the  training  objectives. 

9.  Class  participation  and  interaction  were  encouraged. 

The  results  of  the  surveys  are  shown  in  Figures  4.20  through  4.27.  The  results  indicate  that 
the  subjects  had  a  generally  positive  opinion  of  the  training  as  all  the  surveys  have  as  their 
majority  4’s  and  5’s  for  responses.  Figure  4.20  through  4.22  display  the  results  for  Questions  1 
through  3  respectively.  The  results  indicate  that  the  subjects  felt  that  there  expectations  of  the 
training  were  met.  They  also  showed  confidence  in  their  ability  to  apply  what  they  learned  to 
their  final  day  of  the  experiment.  Figure  4.22  indicates  that  the  subjects  felt  the  information  was 
organized  and  easy  to  follow. 
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Figure  4.20:  Summary  of  Post  Training  Student  Survey  Question  1 
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Chart  of  2 
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Figure  4.21:  Summary  of  Post  Training  Student  Survey  Question  2 


Chart  of  3 
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Figure  4.22:  Summary  of  Post  Training  Student  Survey  Question  3 

As  for  Figures  4.23  through  25,  they  indicate  that  the  subjects  felt  the  information  provided  to 
them  was  useful  and  pertinent  to  what  they  were  studying.  They  had  mostly  high  reviews  for  the 
instructor  proctoring  their  second  day  of  training  and  they  were  generally  pleased  with  the 
quality  of  the  instruction. 
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Chart  of  5 
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Figure  4.23:  Summary  of  Post  Training  Student  Survey  Question  5 
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Figure  4.24:  Summary  of  Post  Training  Student  Survey  Question  6 
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Chart  of  7 
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Figure  4.25:  Summary  of  Post  Training  Student  Survey  Question  7 
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Figure  4.26:  Summary  of  Post  Training  Student  Survey  Question  8 
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Chart  of  9 
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Figure  4.27:  Summary  of  Post  Training  Student  Survey  Question  9 

The  results  for  the  final  two  questions  are  shown  in  Figure  4.26  and  4.27  respectively.  Figure 
4.26  shows  that  the  subjects  felt  the  instructor  adhered  to  the  learning  objectives  provided  during 
the  first  day  of  training.  The  results  shown  on  Figure  4.27  indicate  that  the  subjects  felt  they  were 
allowed  to  participate  in  the  instruction.  Since  this  was  an  individual  study,  it  can  be  assumed 
that  they  interpreted  this  question  as  they  felt  they  were  allowed  to  ask  questions  and  get 
feedback.  The  feedback  provided  though  was  only  to  help  with  problems  understanding  a  word 
or  a  sentence  and  not  with  answering  questions.  The  results  of  the  three  surveys  indicate  that  the 
subjects  had  a  generally  positive  reaction  to  their  training.  In  the  next  section,  the  results  of  how 
they  utilized  their  training  are  analyzed. 

4.4  Day  3 

On  the  final  day  of  training,  each  subject  follows  the  same  procedure.  Their  first  step  is  to 
take  the  State-Trait  Anxiety  Questionnaire.  Following  completion  of  this  questionnaire,  the 
subject  is  taken  to  a  room  to  have  the  physiological  sensors  attached.  After  the  physiological 
equipment  is  attached  to  the  subject,  the  subject  attempts  the  final  scenario  which  consists  of  a 
victim  with  a  mockup  of  an  explosive  device  attached  to  their  back.  The  subject’s  job  is  to  defuse 
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the  device  within  15  minutes  while  perfonning  the  render  safe  procedures  taught  in  the  first  day 
of  training  and  reinforced  during  the  second  day.  The  purpose  of  this  experiment  is  to  determine 
whether  the  instruction  type  had  an  effect  on  the  performance  of  the  subjects  when  it  came  to 
defusing  the  explosive  and  displaying  knowledge  of  the  render  safe  procedures.  However,  before 
answering  this  question,  there  is  interest  in  determining  if  the  test  taken  at  the  conclusion  of  Day 
1  is  predictive  of  how  the  subjects  perfonn  during  Day  3. 

4.4.1  Final  Scenario  Scoring 

The  final  scenario  is  graded  in  three  ways.  One  way  is  while  the  subjects  are  perfonning 
their  final  scenario,  a  grader  marks  whether  the  steps  of  the  render  safe  procedures  are  followed 
correctly.  If  a  subject  performs  a  step,  they  are  given  10  points.  In  total,  there  are  19  render  safe 
procedures  which  corresponds  to  a  maximum  score  of  190  (the  minimum  score  is  0).  A  summary 
of  the  scores  is  displayed  in  Figure  4.28. 
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70.000 

94.393 

95%  Confidence  Interval  for  StDev 

32.090 

45.499 

Figure  4.28:  Summary  for  Render  Safe  Procedure  Score 


The  summary  shows  that  on  average  the  subjects  did  not  perform  very  well  when  it  came  to 


completing  the  render  safe  procedures.  The  average  was  only  a  score  of  83.23  which  is  less  than 


half  of  the  total  available  points.  The  mean  score  is  expected  to  fall  with  95%  confidence  within 
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the  values  of  73.91  and  92.55.  The  upper  level  of  this  confidence  interval  is  still  less  than  half  of 
the  maximum  score.  This  indicates  that  the  training  did  not  do  a  good  job  of  teaching  the  render 
safe  procedures  to  the  subjects. 

The  other  two  scoring  methods  are  linked  together.  The  first  simply  assesses  whether  the 
subjects  properly  defused  the  bomb  or  not  (‘Yes’:  1,  ‘No’:  0).  The  second,  called  the  Defusing 
Score,  is  a  5  point  scale  based  on  what  components  of  the  explosive  the  subject  properly 
disconnected.  The  explosive  used  in  the  final  scenario  consists  of  4  primary  components. 

Radio  transmitter  (walkie-talkie) 

1st  battery 

2nd  battery 

Initiators  (4  blasting  caps) 

When  being  scored,  the  subject  starts  with  a  1  and  gains  a  point  for  every  component  they 
properly  disconnect.  The  key  is  to  disconnect  the  components  in  the  proper  order  which  is  the 
order  the  components  are  listed  in  (the  batteries  can  be  disconnected  in  any  order).  For  example, 
if  a  battery  is  disconnected  prior  to  the  radio  transmitter  being  disconnected  the  subject  receives 
only  a  1  and  does  not  receive  credit  for  any  other  actions  perfonned.  However,  if  the  subject 
disconnects  the  radio  transmitter  but  then  pulls  the  blasting  caps  they  still  get  the  point  for 
disconnecting  the  radio  transmitter.  They  are  not  given  any  points  for  disconnecting  a  battery 
after  they  disconnect  the  initiators.  A  score  of  4  or  5  indicates  that  the  subject  prevented  an 
explosion.  Scores  from  1  to  3  indicate  that  the  explosive  detonated,  and  the  subject  failed  the 
assessment.  Two  bar  charts  in  Figures  4.29  and  4.30  show  a  graphical  representation  of  the  raw 
data  for  these  two  scoring  methods. 
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Chart  of  Diff  Score 


Diff  Score 

Figure  4.30:  Bar  Chart  of  Defusing  Scores 

In  Figure  4.29,  ‘Yes’  and  ‘No’  refer  to  whether  there  was  an  explosion  during  the 
experiment.  As  can  be  seen  the  numbers  are  about  equal  with  the  ‘No’s’  having  a  slight  edge. 
Figure  4.30  indicates  that  of  the  subjects  that  passed,  few  actually  completed  the  final  step  of 
pulling  the  blasting  caps.  Figure  4.30  also  indicates  that  there  were  quite  a  few  subjects  who 
failed  to  properly  perform  any  operations.  The  data  in  these  charts  only  shows  how  the  subjects 
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performed  as  a  whole.  The  purpose  is  to  determine  if  the  instruction  type  the  subjects  received 
affected  their  perfonnance  on  the  final  day.  This  is  to  be  determined  in  the  following  sections. 
4.4.2  Final  Scenario  vs.  Render  Safe  Assessment 

Before  determining  the  effect  of  the  instruction  type  on  the  subjects’  final  day  of 
perfonnance,  a  reference  to  Day  1  is  necessary.  In  section  4.2.2,  the  results  of  the  Render  Safe 
Assessment,  completed  by  each  subject  during  Day  1,  are  summarized.  There  is  interest  in 
detennining  if  success  on  this  assessment  is  predictive  of  success  during  the  final  scenario.  Three 
tests  of  hypothesis,  one  for  each  scoring  technique,  are  used  to  detennine  if  this  is  the  case. 
Render  Safe  Procedure  Score  vs.  Render  Safe  Assessment 
Explosion  (Yes  or  No)  vs.  Render  Safe  Assessment 
Defusing  Scores  vs.  Render  Safe  Assessment 
4.4.2. 1  Render  Safe  Procedure  Score  vs.  Render  Safe  Assessment 
In  order  to  determine  whether  performance  on  the  Render  Safe  Assessment  completed 
during  Day  1  is  predictive  of  success  in  completing  the  render  safe  procedures  tested  for  during 
the  final  day  of  the  experiment,  a  linear  regression  model  is  fitted.  Figure  4.3 1  displays  a  scatter 
plot  of  the  data  with  the  Render  Safe  Procedure  Score  plotted  against  the  subjects’  Render  Safe 


Assessment  score. 
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Scatterplot  of  RSP  Score  vs  RSP  Assessment 
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Figure  4.31:  Scatter  Plot  of  Render  Safe  Procedure  Score  vs.  Render  Safe  Procedure  Assessment 

The  scatter  plot  indicates  that  there  is  not  a  linear  relationship  between  the  two  scores.  In  order  to 
verify,  the  linear  regression  model  is  displayed  below: 

Regression  Analysis:  RSP  Score  versus  RSP  Assessment 

The  regression  equation  is 

RSP  Score  =  66.9  +  0.266  RSP  Assessment 


Predictor  Coef  SE  Coef  T  P 
Constant  66.86  21.26  3.14  0.003 
RSP  Assessment  0.2662  0.3371  0.79  0.433 


S  =  37.7405  R-Sq  =  1.0%  R-Sq(adj)  =  0.0% 


The  regression  analysis  verifies  that  the  Render  Safe  Assessment  is  not  predictive  of 


performance  in  regards  to  displaying  knowledge  of  render  safe  procedures  during  the  final  day  of 


training  as  the  p-value  is  well  above  0.05  for  the  predictor  ‘RSP  Assessment’. 


4.4.2. 2  Explosion  (Yes  or  No)  vs.  Render  Safe  Assessment 

In  order  to  test  whether  performance  on  the  Render  Safe  Assessment  is  predictive  of  the 
subject  properly  defusing  the  bomb  a  different  type  of  regression  is  necessary.  This  is  due  to  the 
fact  that  the  data  for  the  response  variable  ‘Explosion’  is  binary  which  causes  the  assumptions  of 
linear  regression  to  fail.  The  model  used  in  this  case  is  called  Binary  Logistic  Regression  which 
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is  not  limited  by  the  assumption  of  nonnality  and  independence  of  the  residuals.  Instead,  the  only 
assumptions  are  that  the  dependent  variable  is  binary  and  that  the  independent  variable  is  what  is 
called  “linear  in  the  logit.”  Montgomery  et  al.  (2012)  include  a  discussion  on  logistic  regression 
on  pages  421-430  in  their  work  Introduction  to  Linear  Regression  Analysis,  5th  Edition. 

The  binary  logistic  regression  method  fits  the  data  to  the  function 

1 

y  = - - 

1  +  e~xP 

where  (3  is  the  column  vector  of  estimated  coefficients  for  the  linear  part  of  the  equation  and  x  is 
the  row  vector  of  the  predictor  variables.  In  order  to  test  the  linearity  in  the  logit  assumption,  the 
linear  portion  of  the  function  (the  part  contained  in  the  exponent)  is  plotted  against  the 
independent  variable.  If  a  linear  relationship  exists  than  the  assumption  is  validated.  Note  that 
this  only  applies  to  continuous  independent  variables.  Discrete  independent  variables  are  not 
confined  by  this  assumption.  The  results  of  the  binary  logistic  regression  model  are  shown 


below: 
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Link  Function:  Logit 

Response  Information 

Variable  Value  Count 

Explosion_l  Yes  30  (Event) 

No  35 

Total  65 


Logistic  Regression  Table 


Odds 

95% 

Cl 

Predictor 

Coef 

SE  Coef  Z 

P 

Ratio 

Lower 

Upper 

Constant 

-0.344563 

1.13293  -0.30 

0.761 

RSP  Assessment 

0.0030943 

0.0179533  0.17 

0.863 

1.00 

0.97 

1.04 

Log- Likelihood 

=  -44 .847 

Test  that  all 

slopes  are 

zero:  G  =  0.030,  DF 

=  1,  P- 

-Value 

=  0.863 

The  results  show  that  there  is  not  a  relationship  between  whether  the  explosive  detonated  and  the 
results  of  the  Render  Safe  Assessment.  A  similar  test  is  used  to  determine  whether  the  subjects’ 
Render  Safe  Assessment  score  is  predictive  of  the  score  the  subject  receives  from  disconnecting 
particular  components  of  the  explosive  device. 

4.4.2.3  Defusing  Score  vs.  Render  Safe  Assessment 

The  primary  difference  between  the  model  used  for  this  test  and  the  test  in  the  previous 
model  is  that  the  Defusing  Score  is  not  binary.  There  are  five  possible  scores  a  subject  can 
receive.  Since  a  higher  score  is  preferred  over  a  lower  score,  the  response  variable  is  considered 
to  be  ordinal.  Therefore,  an  ordinal  logistic  model  is  used.  The  results  are  shown  below: 
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Link  Function:  Logit 


Response  Information 
Variable  Value  Count 


Diff  Score  1  15 

2  12 

3  3 

4  26 

5  9 

Total  65 


Logistic  Regression  Table 


Odds 

95% 

Cl 

Predictor 

Coef 

SE  Coef 

Z 

P 

Ratio 

Lower 

Upper 

Const  (1) 

-1 . 75957 

1 .05170 

-1  .  67 

0.094 

Const  (2) 

-0 .891404 

1 . 03437 

-0.86 

0.389 

Const (3) 

-0.703641 

1 . 03217 

-0.68 

0.495 

Const  (4) 

1.27814 

1.04806 

1.22 

0.223 

RSP  Assessment 

0.0090765 

0 .0162441 

0.56 

0.576 

1 . 01 

0.98 

1.04 

Log- Likelihood 

=  -92.954 

Test  that  all 

slopes  are 

zero:  G  =  0. 

.320,  DF 

=  1,  P- 

-Value 

=  0.571 

The  results  once  again  indicate  that  the  Render  Safe  Assessment  is  not  predictive  of 
perfonnance.  In  this  case  it  is  not  predictive  of  performance  on  the  Defusing  Score.  The  results 
of  all  the  tests  run  thus  far  indicate  that  the  Render  Safe  Assessment  did  not  have  a  significant 
relationship  with  the  final  assessment  conducted  on  the  final  day  of  the  study.  This  indicates  that 
any  effect  on  the  results  of  the  subjects’  performance  on  the  final  day  would  be  from  the 
instruction  type  and  potentially  the  gender  of  the  victim  in  the  final  assessment. 

4.4.3  Final  Scores  vs.  Instruction  Type  and  Victim’s  Gender 

The  primary  goal  of  the  experiment  is  to  assess  whether  the  instruction  type  had  an  effect 
on  the  final  scores  during  the  final  day  of  the  experiment.  Since  a  subject  could  be  assigned  to 
either  a  female  or  a  male  victim,  Victim’s  Gender  is  included  as  a  potential  predictor  variable  for 
the  model.  The  experiment  consisted  of  65  male  subjects  which  were  separated  into  a  group  of 
32  for  Second  Life  and  33  for  case  studies.  Each  group  was  administered  a  particular  instruction 
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type  (either  Second  Life  or  the  case  studies).  Of  the  65  subjects,  33  were  assigned  to  a  female 
victim  while  32  were  assigned  to  a  male  victim.  The  number  of  male  and  female  victims  for  the 
two  instruction  types  was  approximately  equal. 

4.4.3. 1  Render  Safe  Procedures  Score  vs.  Instruction  Type  and  Victim’s  Gender 
The  first  of  the  scoring  types  is  again  the  Render  Safe  Procedures  Score.  A  matrix  scatter 
plot  in  Figure  4.32  displays  the  relationship  between  the  two  predictor  variables  (Instruction 
Type  and  Victim’s  Gender)  and  the  Render  Safe  Procedures  Score. 


Figure  4.32:  Matrix  Plot  of  Render  Safe  Procedure  Score  vs.  Predictor  Variables 


As  far  as  the  Instruction  Type  is  concerned,  the  means  appear  to  be  located  at  around  the  80  for 
both  instruction  types  (0:  case  studies,  1:  Second  Life).  The  only  difference  is  in  the  dispersion  of 
scores.  For  victim  gender,  the  mean  appears  slightly  higher  for  male  subjects  (1)  than  for  female 
subjects  (0).  A  linear  regression  model  is  used  to  assess  if  any  of  the  predictor  variables 
significantly  affect  the  Render  Safe  Procedures  Score. 
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The  regression  equation  is 

RSP  Score  =  67.4  +  5.34  Instruction  Type  +  26.8  Victim  Gender 


Predictor  Coef  SE  Coef  T  P 
Constant  67.398  7.697  8.76  0.000 
Instruction  Type_l  5.345  8.848  0.60  0.548 
Victim  Gender  1  26.815  8.848  3.03  0.004 


S  =  35.6261  R-Sq  =  13.2%  R-Sq(adj)  =  10.4% 

The  results  indicate  that  the  only  significant  factor  is  the  Victim’s  Gender.  As  such,  the 
Instruction  Type  can  be  eliminated  from  the  model  to  produce  the  model  below: 

Regression  Analysis:  RSP  Score  versus  Victim  Gender 


The  regression  equation  is 

RSP  Score  =  70.2  +  26.6  Victim  Gender 


Predictor  Coef  SE  Coef  T  P 
Constant  70.152  6.170  11.37  0.000 
Victim  Gender  26.567  8.794  3.02  0.004 


S  =  35.4461  R-Sq  =  12.7%  R-Sq(adj)  =  11.3% 

The  low  R2  value  implies  that  there  are  other  variables  accounting  for  the  variance  in  the  scores 
as  the  Victim’s  Gender  only  accounts  for  12.7%.  However,  the  model  still  indicates  that  the 
Victim’s  Gender  had  an  effect  on  the  Render  Safe  Procedures  Score.  Additionally,  the 
coefficient  is  positive  indicating  that  the  subject  did  better  at  perfonning  the  Render  Safe 
Procedures  with  a  male  victim  than  a  female  victim. 

4.4.3.2  Model  Adequacy 

With  a  model  established,  it  is  important  to  test  the  assumptions  of  the  regression  model. 
The  assumptions  of  linear  regression  are  that  the  residuals  are  normally  and  independently 
distributed  with  a  mean  of  zero  and  constant  variance.  The  nonnality  and  zero  mean  assumption 
can  be  detennined  by  using  a  nonnal  probability  plot  of  the  residuals  as  shown  in  Figure  4.33.  It 
should  be  noted  that  all  residuals  used  in  the  adequacy  check  are  standardized  residuals 
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Probability  Plot  of  SRES5 

Normal  -  95%  Cl 


Mean 

-2.90878E-15 

StDev 

1.008 

N 

65 

AD 

0.328 

P-Vaiue 

0.512 

SRES5 


Figure  4.33:  Normal  Probability  Plot  of  Residuals 

Figure  4.33  indicates  that  the  normality  assumption  is  valid  at  that  95%  confidence  level.  As  can 
be  seen  all  of  the  data  points  fit  within  the  confidence  bands  and  the  p-value  is  well  over  0.05 
leading  to  the  conclusion  that  the  hypothesis  that  the  data  fits  the  normal  distribution  fails  to  be 
rejected.  Figure  4.33  also  shows  that  mean  is  very  close  to  zero  which  gives  validity  to  the  zero 
mean  assumption.  As  for  the  constant  variance  assumption,  two  types  of  plots  are  used.  One  of 
the  plots  shown  in  Figure  4.34  is  a  plot  of  the  residuals  vs.  the  fitted  values  of  the  model.  The 
other  plot  is  shown  in  Figure  4.35  and  displays  the  relationship  between  the  residuals  and  the 
predictor  variable,  Victim’s  Gender. 
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Versus  Fits 

(response  is  RSP  Score) 


Figure  4.34:  Residuals  vs.  Fitted  Values 


Residuals  Versus  Victim  Gender 

(response  is  RSP  Score) 


Figure  4.35:  Residuals  vs.  Victim's  Gender 

The  two  plots  do  not  show  any  indication  of  a  pattern  between  the  residuals  and  neither  the 
fitted  values  nor  the  Victim’s  Gender.  The  final  diagnostic  checks  to  see  if  the  independence 
assumption  is  adequate.  This  check  consists  of  checking  for  patterns  in  the  plot  of  the  residuals 
vs.  the  time  sequence  the  data  was  collected.  In  this  experiment,  while  subjects  were  assigned 
numbers  they  were  not  necessarily  tested  in  order  of  those  numbers.  Instead  the  testing  schedule 


72 


95 


was  dictated  by  their  convenience.  Figure  4.36  shows  a  plot  of  the  residuals  vs.  the  order  the  data 
was  recorded. 


Scatterplot  of  SRES5  vs  Order 


Figure  4.36:  Residuals  vs.  Order 

The  graph  in  Figure  4.36  does  not  appear  to  have  any  pattern.  The  data  points  appear  to  be 
random  and  they  do  not  simply  oscillate  about  the  line  y  —  0.  Therefore,  there  does  not  appear  to 
be  a  problem  with  the  independence  assumption.  Thus,  the  assumptions  appear  to  be  validated 
and  the  conclusions  drawn  from  the  linear  model  can  be  properly  inferred.  In  the  next  two 
models,  the  response  variables  are  no  longer  continuous.  Therefore,  logistic  regression  is  used 
once  again. 

4.4.3.3  Explosion  (Yes  or  No)  vs.  Instruction  Type  and  Victim’s  Gender 

In  section  4.4.2,  binary  logistic  regression  was  introduced  as  a  method  for  modeling  data 
with  binary  response  variables.  This  method  is  employed  here  to  detennine  if  the  two  predictor 
variables  have  a  significant  effect  on  whether  the  subject  defused  the  bomb.  A  bar  chart 
displaying  how  subjects  in  particular  groups  performed  is  shown  in  Figure  4.37. 
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Chart  of  Instruction  Type_l,  Victim  Gender_l,  Explosion_l 


Victim  Gender_l  F  M  F  M 

Instruction  Type_l  CS  SL 

Figure  4.37:  Bar  Chart  of  Explosions  by  Victim's  Gender  and  Instruction  Type 

The  chart  shows  that  subjects  taking  the  Second  Life  instruction  type  tended  not  to  have 
explosions  more  so  than  those  taking  the  case  studies.  As  for  Victim’s  Gender,  subjects  taking 
the  case  studies  with  a  male  victim  tended  to  outperform  those  that  took  case  studies  and  had  a 
female  victim.  However,  the  trend  is  reversed  for  the  subjects  using  Second  Life.  Overall  the 
subjects  with  male  victims  outperformed  the  subjects  with  female  victims  by  only  one  successful 
defusing.  Also  of  interest  is:  if  the  length  of  time  the  subject  took  to  complete  the  final 
assessment  had  an  effect  of  whether  they  successfully  defused  the  explosive. 

In  order  to  verify  the  inferences  drawn  from  simply  looking  at  the  graphs,  a  binary 
logistic  regression  analysis  is  performed. 
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Link  Function:  Logit 


Response  Information 

Variable  Value  Count 

Explosion  Yes  30  (Event) 

No  35 

Total  65 


Logistic  Regression  Table 


Odds 

95% 

Cl 

Predictor 

Coef 

SE  Coef 

Z 

p 

Ratio 

Lower 

Upper 

Constant 

Instruction  Type 

0 .434564 

0.444192 

0.98 

0.328 

SL 

Victim  Gender 

-0.967136 

0 .514820 

00 

00 

\ — 1 

1 

0.060 

0.38 

0.14 

1 .04 

M 

-0.248477 

0.514166 

-0.48 

0.629 

0.78 

0.28 

2.14 

Log-Likelihood  =  -42.968 

Test  that  all  slopes  are  zero:  G  =  3.788,  DF  =  2,  P-Value  =  0.150 

The  results  indicate  that  none  of  the  predictor  variables  are  significant  at  the  95%  significance 
level.  However,  the  Instruction  Type  is  close  to  being  considered  significant  as  the  p-value  is 
0.06.  If  the  significance  level  is  set  at  90%,  the  Instruction  Type  is  considered  significant  at  the 
90%  significance  level.  The  results  of  the  test  indicate  that  the  subjects  training  in  Second  Life 
performed  better  than  those  taking  the  case  studies.  Of  note  is  the  odds  ratio  which  represents 
how  likely  the  “Event”  (an  explosion)  occurs  for  one  option  of  a  predictor  variable  relative  to  the 
other.  In  the  case  of  the  Instruction  Type,  the  odds  of  a  subject  taking  Second  Life  failing  to 
defuse  the  bomb  is  0.38  times  that  of  the  odds  of  a  subject  taking  the  case  studies  failing  to 
defuse  the  bomb.  This  shows  that  the  odds  of  an  explosion  are  less  for  the  Second  Life 
instruction  type  than  for  the  case  studies  instruction  type.  However,  the  upper  part  of  the 
confidence  interval  for  the  odds  ratio  is  over  1  which  indicates  that  there  is  a  chance  that  the  odds 
of  an  explosion  for  subjects  taking  Second  Life  can  be  higher  than  the  odds  for  those  taking  the 


case  studies. 
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This  model  analyzes  how  well  the  subjects  performed  from  on  a  pass/fail  basis.  But  just 
as  in  section  4.4.2,  an  ordinal  logistic  model  is  appropriate  for  detennining  how  significantly 
Instruction  Type  and  Victim’s  Gender  affect  the  subjects’  performance  in  terms  of  their  Defusing 
Score. 

4.4.3.4  Defusing  Score  vs.  Instruction  Type  and  Victim’s  Gender 

In  this  model,  the  effect  the  three  predictor  variables  have  on  performance  in  regards  to 
the  subjects’  Defusing  score  is  analyzed.  Figure  4.38  displays  a  bar  graph  with  a  count  of 
subjects  receiving  a  particular  score.  The  bar  graph  is  separated  by  the  Instruction  Type  and 
Victim’s  Gender. 

Chart  of  Instruction  Type,  Victim  Gender,  Defusing  Score 


Defusing  Score  12345  12345  12345  12345 

Victim  Gender  F  M  F  M 

Instruction  Type  CS  SL 

Figure  4.38:  Bar  Chart  of  Defusing  Scores  by  Instruction  Type  and  Victim's  Gender 

Figure  4.38  indicates  that  the  most  instances  of  the  subject  failing  to  defuse  any  component  of 
the  explosive  correctly  fall  within  the  case  where  a  subject  received  the  case  studies  instruction 
type  and  had  a  female  victim.  The  case  with  the  highest  amount  of  successes  at  defusing  is  the 
one  where  a  subject  receives  the  Second  Life  training  and  is  assigned  a  female  victim.  However, 
all  the  maximum  scores  were  attained  when  the  victim  was  male.  Consistent  with  the  previous 
section,  the  subjects  receiving  Second  Life  training  had  a  higher  amount  of  4’s  and  5’s  which 
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constitute  successes.  Conversely,  the  higher  amount  of  l’s,  2’s,  and  3’s  occur  when  the  subject 
receives  the  case  studies.  Ordinal  Logistic  Regression  is  used  to  analyze  this  data  just  as  it  is 
used  to  analyze  whether  the  Render  Safe  Assessment  score  is  predictive  of  success  in  the  final 
part  of  the  experiment.  The  results  of  the  analysis  are  shown  below: 

Ordinal  Logistic  Regression:  Defusing  Sco  versus  Instruction  ,  Victim  Gender 


Link  Function:  Logit 


Response  Information 


Variable  Value  Count 

Defusing  Score  1  15 

2  12 

3  3 

4  26 

5  9 

Total  65 


Logistic  Regression  Table 


Odds 

95% 

Cl 

Predictor 

Coef 

SE  Coef 

Z 

P 

Ratio 

Lower 

Upper 

Const  (1) 

-0 .507047 

0 .416843 

-1.22 

0.224 

Const  (2) 

0.392586 

0 .410015 

0.96 

0.338 

Const (3) 

0.586142 

0 .413494 

1 . 42 

0.156 

Const (4) 

2.66984 

0 . 537472 

4  .  97 

0.000 

Instruction  Type 

SL 

-0 .794209 

0.461914 

-1 . 72 

0.086 

0.45 

0.18 

1 .12 

Victim  Gender 

M 

-0.761026 

0 .461214 

-1 .  65 

0.099 

0.47 

0.19 

1.15 

Log-Likelihood  = 

-90.400 

Test  that  all  slopes  are  zero 

>:  G  =  5. 

429,  DF 

=  2,  P- 

-Value  = 

0.066 

Once  again,  the  test  shows  there  are  no  significant  predictor  variables  at  the  95%  significance 
level.  However,  both  variables  are  significant  at  the  90%  significance  level.  Instruction  Type  has 
a  p-value  of  0.078  and  an  odds  ratio  of  0.44.  This  indicates  that  lower  scores  most  likely  occur 
with  subjects  receiving  the  case  studies  instruction  type.  However,  the  confidence  interval 
encapsulates  the  value  of  1  indicating  that  there  is  a  chance  that  the  chance  of  lower  scores  can 
be  higher  for  subjects  receiving  Second  Life.  As  for  the  Victim’s  Gender,  the  p-value  is  at  the 
very  edge  of  being  considered  significant  at  the  90%  significance  level.  The  odds  ratio  is  0.47 
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which  indicates  lower  scores  are  more  likely  to  occur  with  female  victims.  The  next  chapter 
provides  some  reasoning  for  these  results  and  their  implications.  Before  looking  into  those 
implications,  the  effect  of  the  predictor  variables  on  the  time  to  complete  the  final  scenario  is 
analyzed  in  the  next  section.  The  final  two  sections  contain  the  results  of  the  physiological  data 
and  the  final  survey  taken  by  the  subjects  respectively. 

4.4.4  Time  (min)  vs.  Instruction  Type  and  Victim’s  Gender 

In  addition  to  the  scoring  metrics,  it  is  of  interest  to  determine  if  the  length  of  time  it 
takes  for  the  subject  to  complete  the  experiment  is  affected  by  the  instruction  type  or  the  victim’s 
gender.  A  relationship  between  time  and  either  of  the  two  factors  can  provide  insight  as  to 
whether  one  of  the  two  predictors  affects  the  subjects’  performance  during  the  final  scenario. 

The  time  to  complete  the  experiment  is  measured  from  the  time  the  subject  has  entered  the  room 
with  the  victim  until  the  time  that  the  subject  says  they  have  completed  defusing  the  explosive. 
The  time  is  recorded  (in  minutes)  by  the  victim  who  has  a  clock  near  them  in  order  to  record  the 
time  lapse.  Figure  4.39  shows  a  matrix  plot  displaying  the  graphical  relationship  between  time 
and  the  two  predictors. 


Figure  4.39:  Matrix  Plot  of  Time  (min),  Instruction  Type,  and  Victim's  Gender 
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There  does  not  appear  to  be  a  relationship  between  Instruction  Type  and  Time  (min).  However, 
there  does  appear  to  be  a  relationship  Victim’s  Gender  and  Time  (min).  A  linear  regression 
model  is  used  to  detennine  if  there  is  a  relationship.  The  results  are  shown  below: 

Regression  Analysis:  Time  (min)  versus  Instruction  Type,  Victim  Gender_1 


The  regression  equation  is 


Time  (min)  =  7.15  + 

0.834  In 

struction 

Type  1 

-  1.56 

Predictor 

Coef 

SE  Coef 

T 

P 

Constant 

7.1463 

0.7553 

9.46 

0.000 

Instruction  Type  1 

0.8336 

0.8683 

0.96 

0.341 

Victim  Gender  1 

-1.5605 

0.8683 

-1.80 

0.077 

S  =  3.49615  R-Sq  = 

■  6.5% 

R-Sq (adj ) 

=  3.5% 

The  results  indicate  that  there  is  no  relationship  between  Time  (min)  and  Instruction  Type. 


However,  the  effect  of  Victim’s  Gender  on  Time  (min)  is  significant  at  the  90%  significance 


level.  Since  subjects  with  male  victims  are  coded  with  a  ‘  1  ’  and  the  coefficient  of  Victim’s 


Gender  is  negative,  the  subjects  with  female  victims  took  a  significantly  longer  time  to  complete 


the  final  scenario  than  the  subjects  with  male  victims. 

4.4.5  ECG  Data 


In  Chapter  3,  the  arrangement  for  the  ECG  is  explained.  The  purpose  of  recording  this 
measurement  is  based  on  the  assumption  that  a  better-prepared  subject  would  elicit  lesser  signs 
of  anxiety  when  faced  with  the  final  scenario.  In  the  case  of  this  study,  elevations  in  heart  rate 
are  being  associated  with  higher  levels  of  anxiety.  In  a  study  by  Elwess  and  Vogt  (2005),  they 
found  that  students  performing  commonly  stressful  college  activities  such  as  oral  presentations 
and  written  exams  were  found  to  have  elevated  heart  rates.  Therefore,  elevated  heart  rate  is  used 
as  an  indicator  of  anxiety. 

The  data  used  to  analyze  the  ECG  is  obtained  by  averaging  the  heart  rate  responses  in 
beats-per-minute  (BPM)  during  the  ten  minute  baseline  period  and  then  averaging  the  heart  rate 
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responses  during  the  actual  scenario.  In  this  section,  three  tests  are  performed.  The  first  test 
determines  whether  there  is  a  difference  in  the  baseline  BPM  for  the  two  instruction  types.  A 
difference  would  indicate  that  subjects  from  one  instruction  type  would  be  more  anxious  than  the 
subjects  from  the  other  type.  The  next  thing  to  be  tested  is  whether  there  is  a  difference  in  the 
average  BPM  from  the  baseline  to  the  actual  scenario.  Finally  it  is  of  interest  to  detennine  if  the 
difference  in  BPM  is  affected  by  the  two  predictor  variables:  Instruction  Type  and  Victim’s 
Gender. 

4.4.5, 1  Baseline  Average  Heart  Rate  vs.  Instruction  Type 

The  first  test  is  meant  to  detennine  whether  there  is  a  difference  between  the  two 
instruction  types  in  average  heart  rate  prior  to  the  start  of  the  experiment.  The  results  are  shown 
below: 

Regression  Analysis:  Baseline  versus  Instruction  Type_1 


The  regression  equation  is 

Baseline  =  121  +  3.87  Instruction  Type_l 


Predictor 

Coef 

SE  Coef 

T 

P 

Constant 

121.374 

3.549 

34.20 

0.000 

Instruction 

Type  1 

3.873 

5.058 

0.77 

0.447 

S  =  20.3886 

R-Sq 

=  0.9% 

R-Sq (adj ) 

=  0.0% 

The  results  indicate  that  there  is  no  difference  in  the  baseline  heart  rate  between  the  subjects  of 
each  instruction  type.  Therefore,  it  can  be  concluded  that  neither  instruction  type  reduced  the 
subjects’  anxiety  prior  to  the  start  of  the  experiment. 

4.4.5.2  Difference  in  Baseline  Heart  Rate  and  Scenario  Heart  Rate 

In  order  to  test  if  there  was  a  change  in  average  heart  rate,  a  Paired  t-test  is  used.  The 


result  of  the  Paired  t-test  is  shown  below: 
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Paired  T  for  Avg  Experiment  -  Baseline 


N 

Mean 

StDev 

SE  Mean 

Avg  Experiment 

65 

118 . 

.  71 

12  . 

.74 

1 . 

.  58 

Baseline 

65 

123. 

.28 

20. 

.32 

2. 

.  52 

Difference 

65 

-4  . 

.  57 

16. 

.22 

2. 

.  01 

95%  upper  bound  for  mean  difference:  -1.21 

T-Test  of  mean  difference  =  0  (vs  <  0) :  T-Value  =  -2.27  P-Value  =  0.013 

The  p-value  of  the  test  is  less  than  0.05.  Therefore,  the  conclusion  is  to  reject  the  hypothesis  that 
the  difference  between  the  average  heart  rate  during  the  baseline  and  the  average  heart  rate 
during  the  experiment  is  the  same.  Using  a  one-tailed  test,  the  conclusion  is  that  the  average 
heart  rate  during  the  experiment  was  less  than  it  was  in  the  baseline. 

4.4.5.3  Change  in  Average  Heart  Rate  vs.  Instruction  Type  and  Victim’s  Gender 
The  final  test  determines  if  there  was  a  difference  in  the  change  of  average  heart  rate 
between  the  two  different  instruction  types  and  the  two  victim  genders.  In  order  to  determine  if 
relationships  exist  between  either  of  the  predictor  variables,  a  linear  regression  model  is  used. 
The  result  of  the  model  is  shown  below: 

Regression  Analysis:  Change  versus  Instruction  Type_1,  Victim  Gender_1 

The  regression  equation  is 

Change  =  -  2.48  -  2.88  Instruction  Type_l  -  1.36  Victim  Gender_l 


Predictor  Coef  SE  Coef  T  P 
Constant  -2.479  3.544  -0.70  0.487 
Instruction  Type_l  -2.881  4.075  -0.71  0.482 
Victim  Gender  1  -1.361  4.075  -0.33  0.739 


S  =  16.4059  R-Sq  =  0.9%  R-Sq(adj)  =  0.0% 

The  model  indicates  that  there  is  no  significant  relationship  between  the  change  in  average  heart 
rate  and  the  two  predictors.  Therefore  it  can  be  concluded  that  the  instruction  type  taken  during 
the  second  day  is  not  related  to  the  change  in  heart  rate  during  the  final  day.  Likewise,  the 
victim’s  gender  does  not  have  an  effect  on  the  subjects’  change  in  heart  rate.  While  there  was  not 
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a  difference  between  the  two  predictor  variables  and  the  change  in  heart  rate,  there  may  be  a 
difference  in  how  the  subjects  perceived  their  level  of  stress  during  the  experiment  and  the  two 
instruction  types.  The  next  section  provides  an  analysis  of  the  final  questionnaire. 

4.4.6  Post  Experiment  Questions  -  Anxiety  Questionnaire 

Following  the  conclusion  of  the  experiment,  each  subject  receives  one  final 

questionnaire.  This  questionnaire  is  used  to  analyze  how  the  subjects  perceived  the  scenario  from 
the  final  day  of  the  experiment.  The  questions  asked  are  listed  below: 

1 .  I  felt  extremely  anxious  when  I  was  defusing  the  bomb. 

2.  The  bomb-defusing  situation  seemed  very  artificial  to  me. 

3.  My  attention  was  extremely  focused  when  defusing  the  bomb. 

4.  I  felt  anxious  only  because  I  was  a  subject  in  an  experiment. 

5.  I  thought  the  explosive  device  would  actually  explode. 

6.  The  bomb  defusing  training  helped  me  very  much  to  complete  the  task. 

7.  I  wasn’t  at  all  nervous  when  working  on  the  explosive  device. 

8.  The  electrodes  connected  to  me  caused  me  to  feel  very  nervous. 

9.  After  a  few  minutes,  I  stopped  paying  attention  to  the  electrodes. 

10. 1  felt  the  scenario  training  helped  me  to  be  very  effective  in  defusing  the  bomb. 

Subjects  responded  to  the  questions  using  a  6-point  Likert  scale  with  a  ‘  1  ’  meaning  ‘Very 
Strongly  Disagree’  and  a  ‘6’  meaning  ‘Very  Strongly  Agree.’  The  results  of  the  survey  are 
shown  in  Figures  4.40  through  4.49. 
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Chart  of  1,  Instruction  Type,  Victim  Gender 


9- 


8  ■ 


Figure  4.40:  Results  to  Anxiety  Questionnaire  Question  1 


Chart  of  2,  Instruction  Type,  Victim  Gender 
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Figure  4.41 :  Results  to  Anxiety  Questionnaire  Question  2 
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Chart  of  3,  Instruction  Type,  Victim  Gender 


Instruction  Type  ^  &  O'  &  <$*■  ej* 
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Figure  4.42:  Results  to  Anxiety  Questionnaire  Question  3 

The  results  of  Questions  1  to  3  are  displayed  in  Figures  4.40  to  4.42  respectively.  The 
results  indicate  that  many  subjects  felt  anxious  when  going  through  the  final  scenario.  Most 
subjects  noted  that  the  final  scenario  did  not  seem  artificial  to  them.  Additionally,  the  subjects 
indicated  that  they  were  focused  during  the  scenario  despite  the  victim’s  constant  pleads  for  help 
and  questioning  of  the  subjects’  abilities. 


Chart  of  4,  Instruction  Type,  Victim  Gender 
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Figure  4.43:  Results  to  Anxiety  Questionnaire  Question  4 
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Chart  of  5,  Instruction  Type,  Victim  Gender 
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Figure  4.44:  Results  to  Anxiety  Questionnaire  Question  5 


Chart  of  6,  Instruction  Type,  Victim  Gender 


Figure  4.45:  Results  to  Anxiety  Questionnaire  Question  6 


Figure  4.43  indicates  that  the  subjects  felt  that  their  anxiety  was  not  just  a  result  of  being  part  of 
an  experiment.  However,  as  Figure  4.44  shows,  a  majority  of  the  subjects  did  not  feel  that  the 


mock  explosive  would  actually  detonate.  Figure  4.45  shows  that  a  majority  of  the  subjects  felt 


that  their  training  sufficiently  prepared  them  for  the  scenario. 
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Chart  of  7,  Instruction  Type,  Victim  Gender 


Figure  4.46:  Results  to  Anxiety  Questionnaire  Question  7 
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Chart  of  8,  Instruction  Type,  Victim  Gender 
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Figure  4.47:  Results  to  Anxiety  Questionnaire  Question  8 
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Chart  of  9,  Instruction  Type,  Victim  Gender 
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Figure  4.48:  Results  to  Anxiety  Questionnaire  Question  9 


Figure  4.49:  Results  to  Anxiety  Questionnaire  Question  10 


As  shown  in  Figure  4.46,  a  majority  of  the  subjects  felt  nervous  during  the  scenario.  However,  as 
Figure  4.47  shows,  the  nervousness  was  usually  not  a  result  of  the  sensors.  Figure  4.48  shows  a 


majority  of  the  subjects  stopped  focusing  on  the  sensors  when  they  started  the  scenario.  The  final 


question’s  responses  are  shown  in  Figure  4.49.  In  this  figure,  most  subjects  believed  that  their 


training  the  past  two  days  prepared  them  to  defuse  the  explosive  device. 
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Just  as  with  the  Day  2  surveys,  the  surveys  for  the  final  day  are  generally  positive  and 
reflect  that  the  subjects  experienced  a  level  of  anxiety  as  a  result  of  the  experiment  and  not  just 
from  being  wired  to  physiological  sensors.  It  should  be  noted  that  the  reported  scores  are  broken 
up  by  Instruction  Type  and  Victim’s  Gender.  The  purpose  is  to  not  only  show  how  subjects 
tended  to  report  on  the  aggregate,  but  also  how  they  reported  within  the  different  groups  created 
in  the  experiment.  Table  4.1  contains  the  results  of  using  an  ordinal  logistic  regression  model  to 
analyze  each  question. 


Table  4.1:  Results  of  Anxiety  Questionnaire 


Question 

Instruction  Type 

Victim's  Gender 

P-value 

Odds  Ratio 

95%  LCL 

95%  UCL 

P-value 

Odds  Ratio 

95%  LCL 

95%  UCL 

1 .  I  felt  extremely  anxious  when  I  was  defusing  the  bomb. 

0.236 

0.58 

0.24 

1.24 

0.091 

2.17 

0.88 

5.32 

2.  The  bomb-defusing  situation  seemed  very  artificial  to  me. 

0.033 

2.7 

1.08 

6.72 

0.477 

0.73 

0.30 

1.76 

3 .  My  attention  was  extremely  focused  when  defusing  the  bomb. 

0.373 

0.67 

0.28 

1.61 

0.684 

0.83 

0.35 

2.00 

4.  I  felt  anxious  only  because  I  was  a  subject  in  an  experiment. 

0.524 

0.75 

0.31 

1.82 

0.691 

1.20 

0.49 

2.91 

5.  I  thought  the  explosive  device  would  actually  explode. 

0.528 

0.75 

0.31 

1.82 

0.081 

2.22 

0.91 

5.43 

6.  The  bomb  defusing  training  helped  me  very  much  to  complete  the  task. 

0.419 

0.68 

0.27 

1.73 

0.539 

0.75 

0.30 

1.89 

7.  I  wasn't  at  all  nervous  when  working  on  die  explosive  device. 

0.184 

1.82 

0.75 

4.43 

0.851 

1.09 

0.45 

2.61 

8.  The  electrodes  connected  to  me  caused  me  to  feel  very  nervous. 

0.464 

1.39 

0.58 

3.34 

0.458 

1.39 

0.58 

3.36 

9.  After  a  few  minutes,  I  stopped  paying  attention  to  the  electrodes. 

0.660 

0.82 

0.34 

1.99 

0.944 

1.03 

0.43 

2.50 

10. 1  felt  the  scenario  training  helped  me  to  be  very  effective  in  defusing  the  bomb. 

0.866 

0.93 

0.38 

2.24 

0.583 

1.28 

0.53 

3.10 

The  response  variable  is  the  range  of  scores  from  1-6.  The  predictor  variables  are  the 


Instruction  Type  and  Victim’s  Gender.  For  all  questions,  the  odds  ratio  for  the  Instruction  Type 


refers  to  the  odds,  relative  to  the  case  studies,  of  a  subject  receiving  the  Second  Life  training 


would  disagree  with  the  given  statement.  Likewise,  the  odds  ratio  for  the  Victim’s  Gender  refers 


to  the  odds,  relative  to  the  subjects  with  female  victims,  of  a  subject  with  a  male  victim  would 


disagree  with  the  given  statement.  The  highlighted  cells  in  Table  4.1  indicate  the  significant 


results  at  the  90%  significance  level.  They  indicate  that  subjects  with  female  subjects  tended  to 


report  a  high  sense  of  anxiety  more  often  than  subjects  with  male  victims.  The  same  is  the  case 


when  it  comes  to  reporting  whether  they  thought  the  bomb  would  explode.  The  only  question 


with  a  significant  response  for  the  Instruction  Type  was  Question  2.  Table  4.1  indicates  that 
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subjects  who  received  the  Second  Life  were  almost  three  times  as  likely  to  not  agree  that  the 
situation  seemed  artificial  as  the  subjects  receiving  the  case  studies. 

This  chapter  contains  analysis  for  all  three  days  of  the  experiment.  More  specifically,  it 
contains  the  results  of  the  tests  used  to  determine  the  whether  one  instruction  type  was  more 
effective  than  the  other  as  well  as  the  results  of  the  heart  rate  measurements.  In  the  next  and  final 
chapter,  some  explanations  of  the  results  reported  in  this  chapter  are  provided  as  well  as  possible 
implications  of  the  results  and  some  recommended  best  practices. 
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CHAPTER  V 

CONCLUSION 

5.1  Summary  of  Results 

The  purpose  of  this  experiment  was  to  compare  two  instruction  types  in  regards  to  their 
effectiveness  at  teaching  some  basic  skills  in  properly  defusing  improvised  explosive  devices.  In 
addition,  there  is  interest  in  determining  if  the  gender  of  the  victim  had  any  effect  on  the  subjects' 
ability  to  defuse  the  explosive.  Overall  there  were  eleven  tests  of  hypothesis  performed. 

1 .  RSP  Score  vs.  RSP  Assessment 

2.  Binary  Explosion  vs.  RSP  Assessment 

3.  Defusing  Score  vs.  RSP  Assessment 

4.  RSP  Score  vs.  Instruction  Type  and  Victim’s  Gender 

5.  Binary  Explosion  vs.  Instruction  Type  and  Victim’s  Gender 

6.  Defusing  Score  vs.  Instruction  Type  and  Victim’s  Gender 

7.  Time  vs.  Instruction  Type  and  Victim’s  Gender 

8.  Baseline  Heart  Rate  vs.  Instruction  Type 

9.  Paired  t-test  between  Baseline  Heart  Rate  and  Scenario  Heart  Rate 

10.  Change  in  Heart  Rate  vs.  Instruction  Type  and  Victim’s  Gender 

11.  Questions  from  Anxiety  Questionnaire  vs.  Instruction  Type  and  Victim’s  Gender 

The  eleven  tests  can  be  broken  up  into  four  groups.  The  first  group  consists  of  testing  for  a 
linear  relationship  between  each  of  the  particular  assessment  measures  and  the  RSP  Assessment 
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completed  after  the  first  day  of  training.  The  second  group  consists  of  testing  for  a  linear 
relationship  between  each  of  the  particular  assessment  measures  and  the  three  predictor  variables 
(Instruction  Type,  Victim’s  Gender,  and  Time).  The  third  group  consists  of  analyzing  the  heart 
rate  data.  The  final  group  consists  of  the  results  from  the  Anxiety  Questionnaire. 

5.1.1  Group  1:  RSP  Assessment 

The  results  indicate  that  the  RSP  Assessment  completed  following  the  first  day  of  training  is 
not  predictive  of  the  subjects'  performance  during  the  final  day  of  the  experiment  for  any  of  the 
assessment  measures.  It  can  be  concluded  that  just  because  a  subject  performs  well  on  a  test  does 
not  mean  they  can  be  as  successful  in  the  field.  Conversely,  the  opposite  should  hold  as  well.  Just 
because  a  subject  performs  poorly  on  a  test  does  not  mean  they  would  perform  poorly  in  the 
field.  This  indicates  that  success  would  be  dependent  on  some  other  variable. 

5.1.2  Group  2:  Assessment  Measures 

In  this  experiment,  the  primary  variable  being  tested  is  the  instruction  type  the  subjects 
receive  on  the  second  day  of  training.  The  results  of  the  experiment  show  that  there  is  no 
significant  difference  between  the  two  instruction  types  when  it  comes  to  performance  in  any  of 
the  assessment  measures  at  the  95%  significance  level.  However,  Instruction  Type  is  significant 
at  the  90%  significance  level.  Furthermore,  the  subjects  trained  in  Second  Life  outperformed 
those  trained  using  case  studies  21  successes  to  14.  This  gives  credibility  to  the  idea  that  Second 
Life  is  an  effective  tool  for  learning. 

One  interesting  result  was  that  the  Victim's  Gender  was  found  to  significantly  affect  the 
subjects'  RSP  Score.  In  particular,  subjects  with  a  male  victim  outperformed  subjects  with  a 
female  victim.  This  could  be  an  indication  that  the  subjects'  memory  of  the  render  safe 
procedures  could  have  been  compromised,  if  they  were  adversely  affected  by  the  sight  of  a 
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female  victim.  The  implication  of  this  hypothesis  is  that  subjects  with  male  victims  are  not 
adversely  affected  and  are  able  to  remember  their  tasks  better.  While  Victim’s  Gender  was  not 
found  to  be  significant  when  it  came  to  the  subjects’  ability  to  simply  defuse  the  bomb,  there  was 
a  significant  relationship  at  the  90%  significance  level  between  Victim’s  Gender  and  the 
Defusing  Score.  This  is  due  to  the  fact  that  there  are  two  possible  scores  for  successfully 
defusing  the  explosive.  The  highest  score  of  ‘5’  was  only  attained  by  subjects  with  male  victims. 
Therefore,  subjects  with  female  victims  were  more  likely  to  receive  lower  scores. 

Additionally,  subjects  with  female  victims  tended  to  take  longer  to  complete  the  scenario. 
This  phenomenon  coupled  with  the  fact  that  subjects  with  female  victims  perfonned  worse  at 
remembering  Render  Safe  Procedures  and  at  obtaining  a  high  Defusing  Score  indicates  that 
female  victims  adversely  affected  the  subjects’  perfonnance.  There  was  no  significant  difference 
in  the  time  taken  to  complete  the  final  scenario  between  the  two  instruction  types. 

5.1.3  Group  3:  ECG 

The  results  of  the  ECG  indicate  that  there  was  no  difference  in  baseline  heart  rate 
between  the  two  instruction  types.  In  addition  the  change  in  heart  rate  from  baseline  to  the 
scenario  was  not  found  to  be  affected  by  Instruction  Type  or  Victim’s  Gender.  While  the 
predictor  variables  were  not  found  to  significantly  affect  the  heart  rate,  the  heart  rate  did  change 
from  baseline  to  scenario.  However,  it  unexpectedly  dropped.  The  reasoning  for  this  is  that  the 
subject  may  have  calmed  down  once  the  scenario  began.  While  the  subject  did  not  experience  a 
raise  in  heart  rate,  the  average  baseline  heart  rate  was  really  high.  An  average  adult  has  a  resting 
heart  rate  between  60-100  BPM  (Mann  2011).  The  subjects  had  an  average  heart  rate  of  over  120 
BPM  during  the  base  line.  This  indicates  that  the  subjects  were  experiencing  a  high  level  of 
anxiety  before  starting  the  experiment.  Therefore  the  anxiety  appears  to  be  a  result  of  the 
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subjects  being  in  an  experiment.  The  final  grouping  summarizes  the  results  of  the  Anxiety 
Questionnaire. 

5.1.4  Group  4:  Anxiety  Questionnaire 

The  results  of  the  questionnaire  appear  to  reflect  the  hypothesis  that  the  subjects  were  in 
fact  feeling  a  sense  of  anxiety  during  the  scenario.  However,  the  subjects’  general  feeling  that  the 
anxiety  was  not  simply  a  result  of  being  in  an  experiment  appears  to  be  contradicted  by  the 
results  of  the  heart  rate  measurements.  While  there  was  no  difference  in  the  change  in  heart  rate 
between  either  the  two  instruction  types  or  victim  genders,  there  were  differences  between  the 
two  predictor  variables  when  it  came  to  how  the  subjects  responded  on  the  questionnaire. 
Subjects  with  female  victims  tended  to  report  anxiety  during  the  scenario  more  often  than 
subjects  with  male  victims.  They  also  reported  that  they  thought  the  bomb  would  actually 
explode  more  than  the  subjects  with  male  victims.  These  higher  levels  of  reported  anxiety  and 
fear  of  explosion  could  be  indicative  of  why  the  subjects  with  female  victims  performed  worse 
than  those  with  male  victims.  As  far  as  the  instruction  type  is  concerned,  subjects  receiving 
Second  Life  tended  to  disagree  with  the  assertion  that  the  scenario  seemed  artificial  more  often 
than  the  subjects  receiving  the  case  studies. 

5.2  Implications 

When  it  came  to  Instruction  Type,  there  was  a  significant  difference  found  between  the 
effectiveness  of  the  two  instruction  types  at  the  90%  significance  level.  This  finding  suggests 
that  a  virtual  simulation  can  teach  a  task  more,  or  at  least  as  effectively  as  traditional  paper-and- 
pen  training.  Therefore,  if  an  organization  is  trying  to  decide  whether  to  invest  in  virtual  reality 
for  pedagogical  purposes,  they  have  evidence  to  support  the  notion  that  their  students  can  leam 
what  they  need  to  learn  with  the  same  or  even  better  proficiency  as  they  could  have  learned  with 
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a  traditional  paper-and-pen  study.  There  is  a  potential  drawback  to  using  virtual  reality  and  that 
is  in  the  initial  cost  of  its  implementation.  If  an  organization  wants  to  design  a  virtual  world  in 
order  to  specifically  teach  something  unique  to  their  organization,  the  cost  of  creating  the  world 
can  be  expensive  at  first.  The  cost  of  contracting  developers  to  create  the  software  is  more  than 
printing  out  a  handout  with  instructions.  However,  if  the  end  result  is  that  those  receiving 
training  (employees,  students,  etc.)  learn  better  with  the  virtual  world  then  the  payoff  can  be  seen 
in  fewer  mistakes.  This  leads  to  higher  quality  work  which  in  turn  makes  the  organization  more 
profitable. 

5.3  Best  Practices 

When  conducting  this  experiment,  there  were  some  important  lessons  learned.  The 
primary  lesson  is  to  have  a  properly  updated  computer  lab  available  through  the  duration  of  the 
experiment.  There  were  multiple  instances  when  certain  functions  of  the  training  did  not  work 
properly  due  to  certain  computer  programs  not  being  downloaded.  While  downloads  were 
eventually  able  to  be  made,  it  delayed  the  subjects'  instruction.  Having  an  updated  computer  lab 
is  important  in  order  to  maximize  the  benefit  of  a  virtual  world.  If  the  computers  are  not  properly 
updated  and  maintained,  certain  problems  such  as  rendering  issues  and  freezing  can  occur  which 
occurred  during  the  experiment. 

Data  collection  for  this  experiment  was  conducted  using  the  Blackboard  Learning 
System.  Each  subject  was  assigned  a  separate  profile  corresponding  to  their  subject  number.  The 
data  was  available  to  be  downloaded  by  the  project  leaders  who  were  given  administrator  access. 
However,  an  issue  arose  due  to  the  original  data  collection  settings.  All  of  the  questionnaires  and 
surveys  were  set  to  "Survey"  mode  in  the  data  collection  settings.  As  a  result,  when  trying  to 
collect  the  data,  the  data  was  randomized  making  it  impossible  to  know  to  which  subject  each 
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row  of  data  belonged.  In  addition,  there  were  dummy  profdes  set  up  for  practice  purposes.  The 
scores  from  those  results  would  get  mixed  in  with  the  actual  data  making  it  impossible  to 
detennine  which  data  was  real  and  which  data  was  practice  data.  As  a  result,  the  data  was 
gathered  by  going  into  each  profile  one  by  one  until  all  data  was  properly  gathered.  In  order  to 
avoid  this  problem,  the  data  collection  setting  should  be  set  to  "Test.”  This  allows  the  data  to  be 
associated  with  their  subject  number  in  order  to  properly  analyze  the  data. 

The  final  recommendations  deal  with  the  final  day's  experiment.  When  taking 
physiological  data,  it  is  important  to  keep  outside  factors  such  as  lighting  and  ambient  sound  as 
constant  as  possible.  This  means  turning  on  all  the  lights  in  the  experiment  room  as  well  as 
limiting  access  to  the  room  to  those  directly  involved  in  the  experiment.  A  method  used  in  the 
experiment  that  aided  in  data  collection  for  the  final  day  was  the  implementation  of  a  video 
camera.  The  subjects  were  recorded  during  the  final  day  of  the  experiment  allowing  for  review  of 
the  film  for  any  missed  data.  The  tape  was  not  used  as  the  primary  method  of  data  collection,  but 
it  is  a  useful  tool  in  case  some  data  is  lost  or  accidently  not  recorded.  These  best  practices  serve 
as  recommendations  for  improving  the  experiment. 
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APPENDIX  A:  BIG  FIVE  INVENTORY 

Table  A  1.1:  Big  Five  Inventory 


Subject 

Personality  Trait 

Extraversion 

Agreeableness 

Conscientiousness 

Neuroticism 

Openness 

1 

29 

42 

40 

21 

42 

2 

34 

41 

39 

22 

48 

3 

34 

42 

43 

22 

41 

4 

26 

41 

29 

21 

34 

5 

28 

41 

38 

20 

36 

6 

30 

35 

28 

23 

35 

7 

22 

26 

29 

32 

44 

8 

30 

44 

39 

21 

40 

9 

18 

36 

26 

22 

29 

10 

23 

35 

32 

32 

42 

11 

24 

35 

29 

23 

32 

12 

30 

37 

41 

28 

36 

13 

29 

35 

41 

15 

34 

14 

35 

35 

28 

18 

33 

15 

28 

39 

38 

23 

34 

16 

34 

39 

45 

20 

48 

17 

27 

38 

37 

24 

46 

18 

29 

39 

36 

21 

35 

19 

28 

38 

33 

27 

37 

20 

25 

33 

31 

24 

37 

21 

29 

40 

37 

22 

38 

22 

35 

41 

45 

21 

45 

23 

22 

36 

33 

19 

39 

24 

35 

27 

31 

23 

42 

25 

29 

40 

25 

24 

32 

26 

30 

32 

28 

25 

36 

27 

23 

37 

36 

23 

35 

28 

19 

38 

34 

24 

36 

29 

22 

35 

29 

27 

31 

30 

28 

42 

42 

22 

42 

31 

32 

37 

40 

24 

42 

32 

27 

27 

28 

25 

41 

33 

28 

40 

32 

28 

32 

34 

26 

36 

35 

21 

37 

35 

35 

43 

43 

24 

48 

36 

34 

27 

28 

26 

49 

37 

27 

39 

37 

23 

31 

38 

35 

40 

40 

21 

42 

39 

26 

29 

30 

24 

47 

40 

23 

32 

26 

24 

31 

41 

23 

36 

33 

25 

40 

42 

36 

38 

35 

26 

36 

43 

23 

38 

35 

21 

34 

44 

30 

31 

32 

24 

28 

45 

35 

29 

38 

26 

48 

46 

31 

36 

36 

22 

36 

47 

30 

35 

39 

21 

37 

48 

25 

36 

27 

22 

29 

49 

27 

30 

30 

22 

38 

50 

19 

37 

26 

24 

31 

51 

27 

39 

36 

21 

39 

52 

28 

40 

43 

22 

42 

53 

34 

40 

32 

23 

41 

54 

25 

28 

36 

21 

42 

55 

36 

41 

39 

20 

42 

56 

31 

37 

29 

25 

38 

57 

14 

36 

35 

24 

44 

58 

30 

38 

34 

21 

35 

59 

27 

41 

37 

19 

33 

60 

26 

43 

42 

19 

33 

61 

32 

35 

37 

20 

37 

62 

37 

36 

39 

25 

38 

63 

16 

28 

30 

28 

35 

64 

31 

43 

39 

19 

46 

65 

25 

38 

31 

25 

38 

101 
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APPENDIX  B:  SENSATION  SEEKING  SCALE 

Table  A2.  1 :  Sensation  Seeking  Scale 


Subject 

Question 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

1 

- 

A 

A 

B 

B 

A 

A 

A 

B 

A 

B 

A 

A 

2 

B 

A 

A 

B 

A 

A 

B 

B 

B 

A 

B 

A 

A 

3 

A 

A 

A 

B 

B 

A 

A 

B 

A 

A 

B 

B 

B 

4 

B 

A 

A 

A 

B 

B 

A 

B 

B 

A 

B 

A 

A 

5 

B 

B 

B 

A 

A 

B 

A 

B 

B 

A 

A 

B 

A 

6 

B 

B 

A 

A 

B 

A 

B 

A 

B 

A 

B 

A 

A 

7 

B 

A 

A 

A 

A 

A 

A 

B 

B 

A 

B 

B 

A 

8 

B 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

B 

A 

9 

B 

A 

A 

A 

B 

A 

A 

B 

B 

A 

B 

B 

A 

10 

B 

A 

B 

A 

A 

A 

A 

A 

B 

B 

A 

A 

A 

11 

A 

B 

B 

- 

B 

A 

A 

B 

B 

A 

A 

B 

A 

12 

B 

B 

B 

A 

B 

B 

A 

B 

B 

A 

B 

A 

A 

13 

B 

A 

A 

B 

B 

B 

A 

A 

B 

A 

B 

A 

A 

14 

A 

A 

A 

B 

B 

A 

A 

B 

A 

B 

A 

B 

B 

15 

B 

A 

A 

B 

B 

A 

A 

A 

A 

A 

B 

B 

B 

16 

B 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

A 

A 

17 

B 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

A 

A 

18 

B 

A 

A 

B 

B 

A 

B 

B 

B 

A 

B 

A 

A 

19 

B 

A 

B 

A 

B 

B 

A 

B 

B 

A 

B 

A 

A 

20 

A 

A 

A 

B 

B 

A 

A 

A 

A 

A 

B 

A 

B 

21 

B 

A 

A 

B 

B 

A 

A 

A 

A 

A 

B 

A 

A 

22 

B 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

A 

A 

23 

B 

A 

A 

B 

B 

A 

B 

B 

B 

A 

B 

A 

A 

24 

B 

A 

A 

B 

B 

A 

B 

A 

B 

A 

B 

B 

A 

25 

A 

A 

A 

B 

B 

A 

A 

B 

A 

A 

B 

B 

B 

26 

A 

A 

B 

B 

A 

A 

A 

B 

A 

A 

B 

B 

B 

27 

A 

A 

A 

B 

B 

B 

A 

A 

A 

A 

B 

A 

A 

28 

B 

A 

A 

A 

B 

B 

B 

B 

A 

A 

B 

A 

B 

29 

B 

A 

A 

A 

B 

A 

A 

A 

A 

A 

A 

A 

A 

30 

A 

A 

A 

B 

B 

A 

A 

A 

B 

A 

B 

A 

A 

31 

B 

A 

A 

B 

B 

B 

A 

A 

A 

A 

A 

B 

A 

32 

B 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

B 

A 

33 

B 

A 

A 

A 

B 

A 

A 

B 

A 

A 

B 

A 

A 

34 

B 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

B 

A 

35 

A 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

B 

A 

36 

B 

A 

A 

B 

A 

A 

A 

B 

B 

A 

B 

A 

A 

37 

B 

A 

B 

A 

B 

B 

A 

A 

A 

A 

A 

B 

B 

38 

B 

B 

A 

B 

B 

A 

A 

B 

B 

A 

B 

A 

A 

39 

A 

A 

A 

B 

A 

A 

B 

B 

A 

B 

B 

B 

A 

40 

A 

B 

A 

B 

A 

B 

B 

A 

B 

A 

A 

B 

A 

41 

B 

A 

A 

B 

B 

B 

A 

B 

B 

A 

B 

A 

A 

42 

B 

A 

B 

A 

B 

A 

A 

B 

A 

A 

B 

B 

A 

43 

B 

A 

A 

A 

B 

A 

A 

B 

A 

B 

B 

B 

B 

44 

B 

A 

A 

A 

B 

A 

A 

A 

A 

B 

A 

B 

B 

45 

B 

A 

A 

A 

A 

A 

A 

B 

A 

A 

A 

A 

A 

46 

A 

A 

A 

B 

B 

A 

A 

B 

B 

A 

B 

B 

B 

47 

A 

A 

A 

A 

A 

A 

A 

A 

B 

A 

B 

A 

A 

48 

A 

A 

A 

A 

B 

B 

A 

B 

A 

A 

B 

A 

B 

49 

A 

A 

A 

A 

B 

B 

B 

B 

A 

A 

B 

A 

B 

50 

A 

A 

B 

B 

B 

A 

A 

B 

B 

A 

A 

B 

A 

51 

A 

A 

A 

A 

B 

A 

A 

B 

B 

A 

A 

B 

A 

52 

B 

A 

A 

B 

B 

B 

A 

B 

B 

A 

B 

A 

A 

53 

B 

A 

B 

B 

B 

B 

B 

B 

A 

A 

A 

A 

54 

B 

A 

A 

A 

B 

A 

A 

A 

B 

A 

B 

A 

A 

55 

B 

A 

A 

A 

B 

A 

B 

B 

A 

A 

A 

A 

A 

56 

A 

A 

B 

B 

B 

B 

A 

A 

A 

A 

B 

B 

B 

57 

B 

A 

B 

B 

B 

A 

A 

A 

A 

A 

B 

B 

B 

58 

B 

A 

A 

B 

B 

A 

B 

B 

A 

B 

B 

B 

B 

59 

B 

A 

A 

A 

B 

B 

A 

B 

B 

A 

B 

B 

A 

60 

A 

A 

B 

A 

B 

A 

A 

A 

A 

A 

B 

A 

B 

61 

B 

A 

A 

B 

B 

A 

A 

A 

B 

A 

B 

B 

A 

62 

A 

A 

A 

B 

B 

A 

B 

B 

A 

A 

B 

B 

B 

63 

B 

A 

A 

B 

B 

B 

A 

B 

B 

A 

B 

A 

A 

64 

B 

A 

A 

B 

B 

A 

B 

B 

A 

B 

B 

B 

B 

65 

B 

A 

A 

B 

B 

A 

B 

B 

B 

A 

B 

B 

A 

103 
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APPENDIX  C:  STATE  TRAIT  ANXIETY  QUESTIONNAIRE 

Table  A3.1:  State-Trait  Anxiety  Questionnaire 


State 

Trait 

1 

21 

23 

2 

34 

37 

3 

35 

45 

4 

44 

37 

5 

48 

41 

6 

38 

47 

7 

33 

36 

8 

30 

24 

9 

41 

44 

10 

41 

38 

11 

42 

46 

12 

21 

24 

13 

27 

32 

14 

37 

37 

15 

22 

23 

16 

21 

20 

17 

26 

23 

18 

50 

31 

19 

29 

31 

20 

32 

45 

21 

32 

28 

22 

20 

22 

23 

38 

38 

24 

37 

38 

25 

31 

35 

26 

39 

42 

27 

25 

33 

28 

21 

27 

29 

36 

44 

30 

21 

30 

31 

24 

26 

32 

33 

42 

33 

26 

32 

34 

28 

33 

35 

26 

23 

36 

30 

31 

37 

21 

28 

38 

31 

26 

39 

32 

39 

40 

55 

53 

41 

34 

33 

42 

29 

35 

43 

28 

36 

44 

20 

33 

45 

20 

26 

46 

27 

24 

47 

38 

35 

48 

31 

35 

49 

51 

47 

50 

31 

37 

51 

43 

37 

52 

40 

34 

53 

35 

37 

54 

34 

23 

55 

21 

23 

56 

25 

30 

57 

36 

39 

58 

43 

48 

59 

24 

35 

60 

22 

29 

61 

29 

24 

62 

28 

32 

63 

32.5 

36 

64 

21 

21 

65 

44.5 

45 

105 
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